2024-09-15T02:35:47.584 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T02:35:47.603 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:35:47.680 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: skanta@redhat.com first_in_suite: false job_id: '7905171' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - mons down - mon down - out of quorum sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T02:35:47.680 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T02:35:47.681 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T02:35:47.681 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T02:35:47.682 INFO:teuthology.task.internal:Checking packages... 2024-09-15T02:35:47.704 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T02:35:47.704 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T02:35:47.704 INFO:teuthology.packaging:ref: None 2024-09-15T02:35:47.704 INFO:teuthology.packaging:tag: None 2024-09-15T02:35:47.704 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:35:47.704 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:35:47.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:35:47.850 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:35:47.852 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T02:35:47.858 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T02:35:47.859 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T02:35:47.864 INFO:teuthology.task.internal:Saving configuration 2024-09-15T02:35:47.880 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T02:35:47.887 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T02:35:47.906 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:29:06.280560', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:35:47.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:29:06.276553', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:35:47.946 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:29:06.279173', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:35:47.964 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:29:06.277962', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:35:47.964 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T02:35:47.971 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-15T02:35:47.971 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-15T02:35:47.971 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-15T02:35:47.971 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-15T02:35:47.971 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T02:35:48.059 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f783eb67ac0>, signals=[15]) 2024-09-15T02:35:48.059 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T02:35:48.065 INFO:teuthology.task.internal:Opening connections... 2024-09-15T02:35:48.065 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:35:48.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:48.157 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:35:48.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:48.245 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:35:48.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:48.334 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:35:48.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:35:48.423 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T02:35:48.430 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-09-15T02:35:48.437 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-09-15T02:35:48.437 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="20.04" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:35:48.486 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=focal 2024-09-15T02:35:48.487 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:35:48.487 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-09-15T02:35:48.512 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-09-15T02:35:48.519 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-09-15T02:35:48.519 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Ubuntu" 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:ID=ubuntu 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE=debian 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="20.04" 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:35:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:35:48.569 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:35:48.569 INFO:teuthology.orchestra.run.smithi118.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:35:48.569 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_CODENAME=focal 2024-09-15T02:35:48.569 INFO:teuthology.orchestra.run.smithi118.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:35:48.569 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-09-15T02:35:48.587 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-09-15T02:35:48.594 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-09-15T02:35:48.594 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="Ubuntu" 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:ID=ubuntu 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE=debian 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="20.04" 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:35:48.644 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:35:48.645 INFO:teuthology.orchestra.run.smithi152.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:35:48.645 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_CODENAME=focal 2024-09-15T02:35:48.645 INFO:teuthology.orchestra.run.smithi152.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:35:48.645 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-09-15T02:35:48.662 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-09-15T02:35:48.669 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-09-15T02:35:48.669 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-09-15T02:35:48.719 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Ubuntu" 2024-09-15T02:35:48.719 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:35:48.719 INFO:teuthology.orchestra.run.smithi191.stdout:ID=ubuntu 2024-09-15T02:35:48.719 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE=debian 2024-09-15T02:35:48.719 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="20.04" 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_CODENAME=focal 2024-09-15T02:35:48.720 INFO:teuthology.orchestra.run.smithi191.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:35:48.720 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-09-15T02:35:48.738 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T02:35:48.747 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T02:35:48.753 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T02:35:48.753 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:35:48.755 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:35:48.757 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:35:48.760 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:35:48.766 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T02:35:48.774 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T02:35:48.774 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:35:48.802 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:35:48.808 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:35:48.810 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:35:48.834 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T02:35:48.901 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T02:35:48.915 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T02:35:48.916 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T02:35:48.916 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:35:48.916 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:35:48.917 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:35:48.917 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:35:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:35:48.923 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-15T02:35:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:35:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:35:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:35:48.925 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-15T02:35:48.926 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-15T02:35:48.926 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-09-15T02:35:48.932 INFO:teuthology.orchestra.run.smithi152.stdout:5.4.0-124-generic 2024-09-15T02:35:48.932 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.4.0-124-generic 2024-09-15T02:35:48.932 DEBUG:teuthology.orchestra.run.smithi152:> sudo apt-get clean 2024-09-15T02:35:48.933 INFO:teuthology.orchestra.run.smithi191.stdout:5.4.0-124-generic 2024-09-15T02:35:48.933 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.4.0-124-generic 2024-09-15T02:35:48.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-get clean 2024-09-15T02:35:48.972 INFO:teuthology.orchestra.run.smithi118.stdout:5.4.0-124-generic 2024-09-15T02:35:48.972 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.4.0-124-generic 2024-09-15T02:35:48.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2024-09-15T02:35:48.974 INFO:teuthology.orchestra.run.smithi006.stdout:5.4.0-124-generic 2024-09-15T02:35:48.976 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.4.0-124-generic 2024-09-15T02:35:48.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-09-15T02:35:49.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-get update 2024-09-15T02:35:49.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-15T02:35:49.084 DEBUG:teuthology.orchestra.run.smithi152:> sudo apt-get update 2024-09-15T02:35:49.202 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-09-15T02:35:49.251 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:35:49.342 INFO:teuthology.orchestra.run.smithi152.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:35:49.367 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:35:49.394 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:35:49.395 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:35:49.395 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:35:49.449 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:35:49.454 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:35:49.460 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:35:49.481 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:35:49.482 INFO:teuthology.orchestra.run.smithi191.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:35:49.536 INFO:teuthology.orchestra.run.smithi152.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:35:49.537 INFO:teuthology.orchestra.run.smithi191.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:35:49.554 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:35:49.664 INFO:teuthology.orchestra.run.smithi152.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:35:49.712 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:35:49.724 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:35:49.727 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:35:49.727 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:35:49.728 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:35:49.753 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:35:49.757 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:49.757 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:35:49.764 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:35:49.768 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:35:49.770 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:35:49.770 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:35:49.770 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:35:49.770 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:35:49.775 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:35:49.784 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:35:49.802 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:35:49.819 INFO:teuthology.orchestra.run.smithi152.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:35:49.834 INFO:teuthology.orchestra.run.smithi152.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:35:49.838 INFO:teuthology.orchestra.run.smithi152.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:35:49.838 INFO:teuthology.orchestra.run.smithi152.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:35:49.866 INFO:teuthology.orchestra.run.smithi152.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:35:49.866 INFO:teuthology.orchestra.run.smithi152.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:35:49.868 INFO:teuthology.orchestra.run.smithi152.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:49.869 INFO:teuthology.orchestra.run.smithi152.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:35:49.877 INFO:teuthology.orchestra.run.smithi152.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:35:49.881 INFO:teuthology.orchestra.run.smithi152.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:35:49.882 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:35:49.883 INFO:teuthology.orchestra.run.smithi152.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:35:49.884 INFO:teuthology.orchestra.run.smithi152.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:35:49.884 INFO:teuthology.orchestra.run.smithi152.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:35:49.886 INFO:teuthology.orchestra.run.smithi152.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:35:49.886 INFO:teuthology.orchestra.run.smithi152.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:35:49.889 INFO:teuthology.orchestra.run.smithi152.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:35:49.890 INFO:teuthology.orchestra.run.smithi191.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:35:49.920 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:35:49.925 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:35:49.928 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:35:49.928 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:35:49.943 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:35:49.950 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:35:49.950 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:35:49.954 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:49.955 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:35:49.959 INFO:teuthology.orchestra.run.smithi191.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:35:49.959 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:35:49.966 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:35:49.968 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:35:49.968 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:35:49.968 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:35:49.977 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:35:49.977 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:35:50.024 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:35:50.030 INFO:teuthology.orchestra.run.smithi152.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:35:50.034 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:35:50.038 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:35:50.038 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:35:50.059 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:35:50.059 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:35:50.062 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:50.063 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:35:50.068 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:35:50.076 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:35:50.078 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:35:50.078 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:35:50.078 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:35:50.086 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:35:50.086 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:35:50.087 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:35:50.087 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:35:50.087 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:35:50.103 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:35:50.103 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:35:50.104 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:35:50.104 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:35:50.188 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:35:50.356 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:35:50.369 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:35:50.369 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:35:50.370 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:35:50.414 INFO:teuthology.orchestra.run.smithi152.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:35:50.443 INFO:teuthology.orchestra.run.smithi152.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:35:50.445 INFO:teuthology.orchestra.run.smithi191.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:35:50.445 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:35:50.456 INFO:teuthology.orchestra.run.smithi152.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:35:50.456 INFO:teuthology.orchestra.run.smithi152.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:35:50.459 INFO:teuthology.orchestra.run.smithi191.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:35:50.464 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:35:50.468 INFO:teuthology.orchestra.run.smithi191.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:35:50.468 INFO:teuthology.orchestra.run.smithi191.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:35:50.483 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:50.484 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:35:50.504 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:35:50.535 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:35:50.546 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:35:50.546 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:35:50.546 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:35:50.546 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:35:50.551 INFO:teuthology.orchestra.run.smithi152.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:35:50.552 INFO:teuthology.orchestra.run.smithi152.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:35:50.556 INFO:teuthology.orchestra.run.smithi191.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:35:50.557 INFO:teuthology.orchestra.run.smithi191.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:35:50.564 INFO:teuthology.orchestra.run.smithi191.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:50.565 INFO:teuthology.orchestra.run.smithi191.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:35:50.570 INFO:teuthology.orchestra.run.smithi152.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:50.570 INFO:teuthology.orchestra.run.smithi152.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:35:50.591 INFO:teuthology.orchestra.run.smithi152.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:35:50.605 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:35:50.605 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:35:50.606 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:35:50.607 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:35:50.610 INFO:teuthology.orchestra.run.smithi191.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:35:50.624 INFO:teuthology.orchestra.run.smithi152.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:35:50.625 INFO:teuthology.orchestra.run.smithi191.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:35:50.626 INFO:teuthology.orchestra.run.smithi191.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:35:50.630 INFO:teuthology.orchestra.run.smithi191.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:35:50.630 INFO:teuthology.orchestra.run.smithi191.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:35:50.631 INFO:teuthology.orchestra.run.smithi191.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:35:50.632 INFO:teuthology.orchestra.run.smithi152.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:35:50.633 INFO:teuthology.orchestra.run.smithi152.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:35:50.633 INFO:teuthology.orchestra.run.smithi152.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:35:50.638 INFO:teuthology.orchestra.run.smithi191.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:35:50.639 INFO:teuthology.orchestra.run.smithi191.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:35:50.659 INFO:teuthology.orchestra.run.smithi152.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:35:50.659 INFO:teuthology.orchestra.run.smithi152.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:35:50.660 INFO:teuthology.orchestra.run.smithi152.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:35:50.661 INFO:teuthology.orchestra.run.smithi152.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:35:50.687 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:35:50.687 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:35:50.687 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:35:50.704 INFO:teuthology.orchestra.run.smithi191.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:35:50.704 INFO:teuthology.orchestra.run.smithi191.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:35:50.733 INFO:teuthology.orchestra.run.smithi191.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:35:50.734 INFO:teuthology.orchestra.run.smithi191.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:35:50.741 INFO:teuthology.orchestra.run.smithi152.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:35:50.741 INFO:teuthology.orchestra.run.smithi152.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:35:50.742 INFO:teuthology.orchestra.run.smithi152.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:35:50.742 INFO:teuthology.orchestra.run.smithi152.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:35:50.746 INFO:teuthology.orchestra.run.smithi191.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:35:50.746 INFO:teuthology.orchestra.run.smithi191.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:35:50.777 INFO:teuthology.orchestra.run.smithi191.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:35:50.804 INFO:teuthology.orchestra.run.smithi191.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:35:50.810 INFO:teuthology.orchestra.run.smithi191.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:35:50.814 INFO:teuthology.orchestra.run.smithi191.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:35:50.815 INFO:teuthology.orchestra.run.smithi191.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:35:50.872 INFO:teuthology.orchestra.run.smithi191.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:35:50.873 INFO:teuthology.orchestra.run.smithi191.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:35:50.873 INFO:teuthology.orchestra.run.smithi191.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:35:50.874 INFO:teuthology.orchestra.run.smithi191.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:35:50.875 INFO:teuthology.orchestra.run.smithi191.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:35:50.954 INFO:teuthology.orchestra.run.smithi191.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:35:50.954 INFO:teuthology.orchestra.run.smithi191.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:35:50.955 INFO:teuthology.orchestra.run.smithi191.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:35:54.221 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 21.8 MB in 3s (7,236 kB/s) 2024-09-15T02:35:54.227 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 21.8 MB in 3s (7,028 kB/s) 2024-09-15T02:35:54.249 INFO:teuthology.orchestra.run.smithi152.stdout:Fetched 21.8 MB in 3s (7,078 kB/s) 2024-09-15T02:35:55.005 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 21.8 MB in 4s (6,021 kB/s) 2024-09-15T02:35:55.753 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-15T02:35:55.779 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:35:55.827 INFO:teuthology.orchestra.run.smithi152.stdout:Reading package lists... 2024-09-15T02:35:55.838 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-15T02:35:55.850 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-15T02:35:55.853 DEBUG:teuthology.orchestra.run.smithi152:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:35:55.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:35:55.925 INFO:teuthology.orchestra.run.smithi152.stdout:Reading package lists... 2024-09-15T02:35:55.961 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-15T02:35:56.068 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-09-15T02:35:56.069 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-09-15T02:35:56.121 INFO:teuthology.orchestra.run.smithi152.stdout:Building dependency tree... 2024-09-15T02:35:56.121 INFO:teuthology.orchestra.run.smithi152.stdout:Reading state information... 2024-09-15T02:35:56.188 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-15T02:35:56.189 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-15T02:35:56.275 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2024-09-15T02:35:56.276 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:56.276 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:35:56.276 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:56.277 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2024-09-15T02:35:56.277 INFO:teuthology.orchestra.run.smithi118.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:35:56.277 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2024-09-15T02:35:56.277 INFO:teuthology.orchestra.run.smithi118.stdout: thermald 2024-09-15T02:35:56.319 INFO:teuthology.orchestra.run.smithi152.stdout:The following additional packages will be installed: 2024-09-15T02:35:56.320 INFO:teuthology.orchestra.run.smithi152.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:56.320 INFO:teuthology.orchestra.run.smithi152.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:35:56.320 INFO:teuthology.orchestra.run.smithi152.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:56.321 INFO:teuthology.orchestra.run.smithi152.stdout:Suggested packages: 2024-09-15T02:35:56.321 INFO:teuthology.orchestra.run.smithi152.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:35:56.321 INFO:teuthology.orchestra.run.smithi152.stdout:Recommended packages: 2024-09-15T02:35:56.321 INFO:teuthology.orchestra.run.smithi152.stdout: thermald 2024-09-15T02:35:56.369 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2024-09-15T02:35:56.369 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:56.370 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:35:56.370 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:56.371 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages will be upgraded: 2024-09-15T02:35:56.371 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:35:56.382 INFO:teuthology.orchestra.run.smithi152.stdout:The following NEW packages will be installed: 2024-09-15T02:35:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:35:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:56.384 INFO:teuthology.orchestra.run.smithi152.stdout:The following packages will be upgraded: 2024-09-15T02:35:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:35:56.404 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-09-15T02:35:56.404 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:56.405 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:35:56.405 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:56.406 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-09-15T02:35:56.406 INFO:teuthology.orchestra.run.smithi006.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:35:56.406 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-09-15T02:35:56.406 INFO:teuthology.orchestra.run.smithi006.stdout: thermald 2024-09-15T02:35:56.464 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2024-09-15T02:35:56.474 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-09-15T02:35:56.474 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:56.474 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:35:56.474 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:56.475 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2024-09-15T02:35:56.476 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:35:56.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:35:56.529 INFO:teuthology.orchestra.run.smithi006.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:35:56.529 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:35:56.529 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:35:56.530 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:35:56.552 INFO:teuthology.orchestra.run.smithi118.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:35:56.552 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:35:56.552 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:35:56.552 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:35:56.566 INFO:teuthology.orchestra.run.smithi152.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:35:56.566 INFO:teuthology.orchestra.run.smithi152.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:35:56.566 INFO:teuthology.orchestra.run.smithi152.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:35:56.566 INFO:teuthology.orchestra.run.smithi152.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:35:56.579 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2024-09-15T02:35:56.798 INFO:teuthology.orchestra.run.smithi191.stdout:Building dependency tree... 2024-09-15T02:35:56.798 INFO:teuthology.orchestra.run.smithi191.stdout:Reading state information... 2024-09-15T02:35:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:35:56.945 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:35:57.003 INFO:teuthology.orchestra.run.smithi191.stdout:The following additional packages will be installed: 2024-09-15T02:35:57.003 INFO:teuthology.orchestra.run.smithi191.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:57.003 INFO:teuthology.orchestra.run.smithi191.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:35:57.004 INFO:teuthology.orchestra.run.smithi191.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:Suggested packages: 2024-09-15T02:35:57.005 INFO:teuthology.orchestra.run.smithi191.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:35:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:Recommended packages: 2024-09-15T02:35:57.005 INFO:teuthology.orchestra.run.smithi191.stdout: thermald 2024-09-15T02:35:57.057 INFO:teuthology.orchestra.run.smithi191.stdout:The following NEW packages will be installed: 2024-09-15T02:35:57.057 INFO:teuthology.orchestra.run.smithi191.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:35:57.057 INFO:teuthology.orchestra.run.smithi191.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:35:57.057 INFO:teuthology.orchestra.run.smithi191.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:35:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:The following packages will be upgraded: 2024-09-15T02:35:57.059 INFO:teuthology.orchestra.run.smithi191.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:35:57.242 INFO:teuthology.orchestra.run.smithi191.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:35:57.242 INFO:teuthology.orchestra.run.smithi191.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:35:57.242 INFO:teuthology.orchestra.run.smithi191.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:35:57.242 INFO:teuthology.orchestra.run.smithi191.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:35:57.412 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:35:57.412 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:35:57.413 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:35:57.508 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:35:57.519 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:35:57.701 INFO:teuthology.orchestra.run.smithi152.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:35:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:35:58.012 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:35:58.020 INFO:teuthology.orchestra.run.smithi152.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:35:58.236 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 77.2 MB in 1s (74.5 MB/s) 2024-09-15T02:35:58.410 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:35:58.410 INFO:teuthology.orchestra.run.smithi191.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:35:58.683 INFO:teuthology.orchestra.run.smithi191.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:35:59.318 INFO:teuthology.orchestra.run.smithi152.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:35:59.319 INFO:teuthology.orchestra.run.smithi152.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:35:59.319 INFO:teuthology.orchestra.run.smithi152.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:35:59.491 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:35:59.491 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:35:59.491 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:35:59.603 INFO:teuthology.orchestra.run.smithi152.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:35:59.617 INFO:teuthology.orchestra.run.smithi152.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:35:59.767 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:35:59.796 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:36:00.034 INFO:teuthology.orchestra.run.smithi191.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:36:00.035 INFO:teuthology.orchestra.run.smithi191.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:36:00.035 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:36:00.161 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:36:00.166 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:00.266 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:00.284 INFO:teuthology.orchestra.run.smithi152.stdout:Fetched 77.2 MB in 3s (23.9 MB/s) 2024-09-15T02:36:00.318 INFO:teuthology.orchestra.run.smithi191.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:36:00.332 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:36:00.473 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:36:00.527 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 77.2 MB in 3s (22.6 MB/s) 2024-09-15T02:36:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:36:00.993 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 77.2 MB in 3s (23.6 MB/s) 2024-09-15T02:36:01.162 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:36:02.175 INFO:teuthology.orchestra.run.smithi152.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:36:02.180 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:02.191 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:36:02.205 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:02.270 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:02.320 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:02.604 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:36:02.609 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:02.753 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:02.917 INFO:teuthology.orchestra.run.smithi191.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:36:02.922 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:03.000 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:03.419 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:36:03.433 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:03.486 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:04.194 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:36:04.209 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:04.308 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:04.805 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:36:04.821 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:04.867 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:36:04.883 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:04.905 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:04.978 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:05.380 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:36:05.394 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:05.438 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:06.041 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:36:06.055 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:06.069 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:36:06.084 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:06.099 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:06.127 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:08.407 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:08.542 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:08.910 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:09.054 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:09.357 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:36:09.371 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:36:09.407 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:10.280 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:10.423 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:10.768 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:10.893 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:10.898 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:11.022 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:11.135 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:11.246 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:36:11.259 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:11.260 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:36:11.304 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:11.357 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:11.484 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:11.595 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:11.722 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:11.812 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:36:11.826 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:36:11.887 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:12.067 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:36:12.083 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:36:12.125 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:14.789 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:36:14.809 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:14.846 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:16.513 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:36:16.535 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:16.579 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:17.265 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:36:17.287 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:17.332 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:17.678 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:17.808 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:17.842 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:36:17.864 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:36:17.909 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:18.175 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:19.405 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:20.046 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:20.157 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:20.174 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:20.309 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:20.499 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:20.594 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:20.595 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:20.605 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:36:20.645 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:20.733 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:20.734 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:36:21.063 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:21.716 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:21.870 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:21.959 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:22.102 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:36:22.228 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:22.354 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:36:22.513 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:36:22.640 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:22.724 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:36:22.724 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:22.730 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:22.855 INFO:teuthology.orchestra.run.smithi191.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:22.855 INFO:teuthology.orchestra.run.smithi191.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:23.018 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:23.019 INFO:teuthology.orchestra.run.smithi152.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:23.019 INFO:teuthology.orchestra.run.smithi152.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:23.164 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:23.885 INFO:teuthology.orchestra.run.smithi118.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:23.885 INFO:teuthology.orchestra.run.smithi118.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:24.057 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:24.195 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:24.321 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:36:24.365 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:24.447 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:24.491 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:36:24.573 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:36:24.617 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:24.698 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:36:24.743 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:36:24.834 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:24.869 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:36:24.901 INFO:teuthology.orchestra.run.smithi191.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:36:24.901 INFO:teuthology.orchestra.run.smithi191.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:25.254 INFO:teuthology.orchestra.run.smithi152.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:25.259 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:36:25.401 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:36:25.527 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:25.653 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:36:25.779 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:36:25.922 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:36:26.001 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:36:26.001 INFO:teuthology.orchestra.run.smithi118.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:26.148 INFO:teuthology.orchestra.run.smithi152.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:36:26.148 INFO:teuthology.orchestra.run.smithi152.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:45.731 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:36:45.732 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:36:45.740 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:45.762 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2024-09-15T02:36:46.128 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:46.145 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:46.510 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:46.521 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:46.764 INFO:teuthology.orchestra.run.smithi191.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:36:46.764 INFO:teuthology.orchestra.run.smithi191.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:36:46.769 INFO:teuthology.orchestra.run.smithi191.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:46.781 INFO:teuthology.orchestra.run.smithi191.stdout:Generating grub configuration file ... 2024-09-15T02:36:47.072 INFO:teuthology.orchestra.run.smithi191.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:47.095 INFO:teuthology.orchestra.run.smithi191.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:47.486 INFO:teuthology.orchestra.run.smithi191.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:47.497 INFO:teuthology.orchestra.run.smithi191.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:36:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:36:48.321 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:48.333 INFO:teuthology.orchestra.run.smithi118.stdout:Generating grub configuration file ... 2024-09-15T02:36:48.527 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-09-15T02:36:48.659 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:48.684 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:48.876 INFO:teuthology.orchestra.run.smithi152.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:36:48.876 INFO:teuthology.orchestra.run.smithi152.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:36:48.882 INFO:teuthology.orchestra.run.smithi152.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:48.893 INFO:teuthology.orchestra.run.smithi152.stdout:Generating grub configuration file ... 2024-09-15T02:36:48.954 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-09-15T02:36:49.011 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-09-15T02:36:49.011 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-09-15T02:36:49.011 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-09-15T02:36:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-09-15T02:36:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 20 2024-09-15T02:36:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-09-15T02:36:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-09-15T02:36:49.012 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:49.013 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:49.013 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:49.013 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-09-15T02:36:49.013 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-09-15T02:36:49.014 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:36:49.014 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:36:49.014 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:36:49.014 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T02:36:49.015 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:36:49.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-09-15T02:36:49.064 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:49.075 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:49.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-15T02:36:49.227 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:36:49.227 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:36:49.233 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:36:49.251 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:36:49.275 INFO:teuthology.orchestra.run.smithi152.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:49.299 INFO:teuthology.orchestra.run.smithi152.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:49.680 INFO:teuthology.orchestra.run.smithi152.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:49.691 INFO:teuthology.orchestra.run.smithi152.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:50.183 INFO:teuthology.orchestra.run.smithi191.stdout:done 2024-09-15T02:36:50.613 DEBUG:teuthology.orchestra.run.smithi191:> dpkg -s linux-image-generic 2024-09-15T02:36:50.659 INFO:teuthology.orchestra.run.smithi191.stdout:Package: linux-image-generic 2024-09-15T02:36:50.659 INFO:teuthology.orchestra.run.smithi191.stdout:Status: install ok installed 2024-09-15T02:36:50.659 INFO:teuthology.orchestra.run.smithi191.stdout:Priority: optional 2024-09-15T02:36:50.659 INFO:teuthology.orchestra.run.smithi191.stdout:Section: kernel 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Installed-Size: 20 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: amd64 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Source: linux-meta 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Recommends: thermald 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: available. 2024-09-15T02:36:50.660 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:36:50.661 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:36:50.661 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:36:50.661 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-15T02:36:50.661 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:36:50.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-get clean 2024-09-15T02:36:50.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-get update 2024-09-15T02:36:50.908 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:36:51.024 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:36:51.106 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:36:51.152 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-15T02:36:51.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:36:51.187 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:36:51.254 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-15T02:36:51.479 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-15T02:36:51.480 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-15T02:36:51.719 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:36:51.719 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:36:51.721 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-09-15T02:36:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-09-15T02:36:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 20 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:51.745 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:51.746 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-09-15T02:36:51.746 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-15T02:36:51.750 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.m0pRlZBsev 2024-09-15T02:36:51.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.m0pRlZBsev 2024-09-15T02:36:51.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.m0pRlZBsev 2024-09-15T02:36:51.852 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.m0pRlZBsev is 9KB 2024-09-15T02:36:51.865 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.m0pRlZBsev 2024-09-15T02:36:51.868 INFO:teuthology.orchestra.run.smithi118.stdout:done 2024-09-15T02:36:51.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:51.884 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:36:51.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:36:51.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:51.960 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:36:51.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2024-09-15T02:36:52.169 INFO:teuthology.orchestra.run.smithi152.stdout:done 2024-09-15T02:36:52.341 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2024-09-15T02:36:52.393 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2024-09-15T02:36:52.394 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2024-09-15T02:36:52.394 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2024-09-15T02:36:52.394 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2024-09-15T02:36:52.394 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 20 2024-09-15T02:36:52.394 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2024-09-15T02:36:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2024-09-15T02:36:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2024-09-15T02:36:52.396 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:52.396 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:52.396 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2024-09-15T02:36:52.396 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:36:52.396 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:36:52.397 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:36:52.397 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T02:36:52.397 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:36:52.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2024-09-15T02:36:52.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-09-15T02:36:52.597 DEBUG:teuthology.orchestra.run.smithi152:> dpkg -s linux-image-generic 2024-09-15T02:36:52.655 INFO:teuthology.orchestra.run.smithi152.stdout:Package: linux-image-generic 2024-09-15T02:36:52.655 INFO:teuthology.orchestra.run.smithi152.stdout:Status: install ok installed 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Priority: optional 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Section: kernel 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Installed-Size: 20 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: amd64 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Source: linux-meta 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Recommends: thermald 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:52.656 INFO:teuthology.orchestra.run.smithi152.stdout: available. 2024-09-15T02:36:52.657 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:36:52.657 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:36:52.657 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:36:52.657 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-15T02:36:52.657 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:36:52.657 DEBUG:teuthology.orchestra.run.smithi152:> sudo apt-get clean 2024-09-15T02:36:52.748 DEBUG:teuthology.orchestra.run.smithi152:> sudo apt-get update 2024-09-15T02:36:52.749 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:36:52.749 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:36:52.776 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2024-09-15T02:36:52.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:36:52.819 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:36:52.871 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2024-09-15T02:36:52.895 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:36:52.900 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:36:52.901 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:36:52.919 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:36:53.012 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:36:53.113 INFO:teuthology.orchestra.run.smithi191.stdout:Building dependency tree... 2024-09-15T02:36:53.113 INFO:teuthology.orchestra.run.smithi191.stdout:Reading state information... 2024-09-15T02:36:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:36:53.332 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:53.359 INFO:teuthology.orchestra.run.smithi191.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:36:53.359 INFO:teuthology.orchestra.run.smithi191.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:36:53.360 DEBUG:teuthology.orchestra.run.smithi191:> dpkg -s linux-image-generic 2024-09-15T02:36:53.380 INFO:teuthology.orchestra.run.smithi191.stdout:Package: linux-image-generic 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Status: install ok installed 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Priority: optional 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Section: kernel 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Installed-Size: 20 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: amd64 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Source: linux-meta 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Recommends: thermald 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:53.381 INFO:teuthology.orchestra.run.smithi191.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:53.382 INFO:teuthology.orchestra.run.smithi191.stdout: available. 2024-09-15T02:36:53.382 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-15T02:36:53.386 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-09-15T02:36:53.387 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.p9E9vZj5mA 2024-09-15T02:36:53.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub/grub.cfg /tmp/tmp.p9E9vZj5mA 2024-09-15T02:36:53.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.p9E9vZj5mA 2024-09-15T02:36:53.498 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.p9E9vZj5mA is 9KB 2024-09-15T02:36:53.512 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.p9E9vZj5mA 2024-09-15T02:36:53.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:53.574 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:36:53.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:36:53.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:53.616 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:36:53.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo update-grub 2024-09-15T02:36:53.726 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:54.152 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:54.165 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:54.623 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-15T02:36:54.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:36:54.721 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-15T02:36:54.741 INFO:teuthology.orchestra.run.smithi152.stdout:Reading package lists... 2024-09-15T02:36:54.767 DEBUG:teuthology.orchestra.run.smithi152:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:36:54.839 INFO:teuthology.orchestra.run.smithi152.stdout:Reading package lists... 2024-09-15T02:36:54.940 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-09-15T02:36:54.941 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-09-15T02:36:55.034 INFO:teuthology.orchestra.run.smithi152.stdout:Building dependency tree... 2024-09-15T02:36:55.035 INFO:teuthology.orchestra.run.smithi152.stdout:Reading state information... 2024-09-15T02:36:55.062 INFO:teuthology.orchestra.run.smithi191.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:36:55.066 INFO:teuthology.orchestra.run.smithi191.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-09-15T02:36:55.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-09-15T02:36:55.081 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-09-15T02:36:55.180 INFO:teuthology.orchestra.run.smithi118.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:36:55.181 INFO:teuthology.orchestra.run.smithi118.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:36:55.182 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2024-09-15T02:36:55.199 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2024-09-15T02:36:55.199 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2024-09-15T02:36:55.199 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2024-09-15T02:36:55.199 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 20 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:55.200 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:55.201 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2024-09-15T02:36:55.201 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-09-15T02:36:55.206 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.KzuH6RTNyj 2024-09-15T02:36:55.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub/grub.cfg /tmp/tmp.KzuH6RTNyj 2024-09-15T02:36:55.217 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.KzuH6RTNyj 2024-09-15T02:36:55.259 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.KzuH6RTNyj is 9KB 2024-09-15T02:36:55.269 INFO:teuthology.orchestra.run.smithi152.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:36:55.269 INFO:teuthology.orchestra.run.smithi152.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:36:55.271 DEBUG:teuthology.orchestra.run.smithi152:> dpkg -s linux-image-generic 2024-09-15T02:36:55.273 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.KzuH6RTNyj 2024-09-15T02:36:55.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Package: linux-image-generic 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Status: install ok installed 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Priority: optional 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Section: kernel 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Installed-Size: 20 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: amd64 2024-09-15T02:36:55.294 INFO:teuthology.orchestra.run.smithi152.stdout:Source: linux-meta 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout:Version: 5.4.0.195.193 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout:Recommends: thermald 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout:Description: Generic Linux kernel image 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:36:55.295 INFO:teuthology.orchestra.run.smithi152.stdout: available. 2024-09-15T02:36:55.295 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-15T02:36:55.337 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:36:55.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:36:55.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:55.340 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.Rj2De2Jtjt 2024-09-15T02:36:55.340 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Rj2De2Jtjt 2024-09-15T02:36:55.344 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:55.356 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:55.364 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:36:55.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo update-grub 2024-09-15T02:36:55.365 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.Rj2De2Jtjt 2024-09-15T02:36:55.404 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.Rj2De2Jtjt is 9KB 2024-09-15T02:36:55.416 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.Rj2De2Jtjt 2024-09-15T02:36:55.422 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:55.480 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:36:55.480 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:36:55.480 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:36:55.511 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:36:55.511 DEBUG:teuthology.orchestra.run.smithi152:> sudo update-grub 2024-09-15T02:36:55.710 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:55.722 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:56.855 INFO:teuthology.orchestra.run.smithi152.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:36:56.863 INFO:teuthology.orchestra.run.smithi152.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:56.869 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:36:56.878 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:36:56.890 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-09-15T02:36:56.905 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-09-15T02:36:57.141 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:57.153 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:57.177 INFO:teuthology.orchestra.run.smithi152.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:36:57.188 INFO:teuthology.orchestra.run.smithi152.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:36:57.331 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-09-15T02:36:57.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-09-15T02:36:57.538 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:57.550 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:57.659 INFO:teuthology.orchestra.run.smithi152.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:36:57.671 INFO:teuthology.orchestra.run.smithi152.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:36:59.117 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-09-15T02:36:59.119 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-09-15T02:36:59.127 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-09-15T02:36:59.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-09-15T02:37:25.097 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T02:37:25.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-15T02:37:25.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:27.337 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-15T02:37:27.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-15T02:37:27.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:29.124 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-15T02:37:29.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:37:29.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:29.132 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T02:37:29.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:37:29.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:43.617 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-15T02:37:45.661 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-09-15T02:37:47.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-15T02:37:47.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-15T02:37:52.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-15T02:37:52.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:54.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-15T02:37:54.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:56.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:37:56.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:56.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:37:56.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:37:57.725 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-09-15T02:37:59.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-15T02:37:59.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-15T02:38:09.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-15T02:38:09.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:38:11.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:38:11.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:38:11.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:38:11.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:38:14.593 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-15T02:38:29.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:38:29.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:38:29.953 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-15T02:38:41.471 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-15T02:38:42.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-15T02:38:42.018 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.4.0-195-generic"... 2024-09-15T02:38:42.018 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-15T02:38:42.024 INFO:teuthology.orchestra.run.smithi191.stdout:5.4.0-195-generic 2024-09-15T02:38:42.025 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:38:42.025 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:38:42.025 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:38:43.025 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-15T02:38:43.026 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:38:43.126 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-09-15T02:38:43.150 DEBUG:teuthology.parallel:result is None 2024-09-15T02:38:44.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:38:44.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:38:45.404 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-15T02:38:46.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:38:46.470 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.4.0-195-generic"... 2024-09-15T02:38:46.470 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-15T02:38:46.477 INFO:teuthology.orchestra.run.smithi152.stdout:5.4.0-195-generic 2024-09-15T02:38:46.477 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:38:46.477 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:38:46.477 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:38:47.477 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-15T02:38:47.478 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:38:47.577 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-09-15T02:38:47.607 DEBUG:teuthology.parallel:result is None 2024-09-15T02:38:52.624 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T02:39:01.413 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-15T02:39:02.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:39:02.011 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-15T02:39:02.011 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-09-15T02:39:02.018 INFO:teuthology.orchestra.run.smithi118.stdout:5.4.0-195-generic 2024-09-15T02:39:02.018 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:39:02.018 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:39:02.018 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:39:03.019 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T02:39:03.019 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:39:03.109 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-09-15T02:39:03.130 DEBUG:teuthology.parallel:result is None 2024-09-15T02:39:04.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-15T02:39:04.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:39:04.868 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-15T02:39:05.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-15T02:39:05.541 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-15T02:39:05.541 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-15T02:39:05.548 INFO:teuthology.orchestra.run.smithi006.stdout:5.4.0-195-generic 2024-09-15T02:39:05.548 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:39:05.548 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:39:05.548 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:39:06.549 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T02:39:06.549 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:39:06.642 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-09-15T02:39:06.674 DEBUG:teuthology.parallel:result is None 2024-09-15T02:39:06.675 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T02:39:06.684 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T02:39:06.684 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:39:06.686 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:39:06.688 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:39:06.691 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:39:06.698 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T02:39:06.704 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T02:39:06.710 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T02:39:06.710 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:39:06.712 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:39:06.714 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:39:06.716 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:39:06.757 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T02:39:06.764 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T02:39:06.764 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:39:06.770 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:39:06.772 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:39:06.774 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:39:06.785 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.792 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.796 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.796 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.801 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.806 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.807 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.817 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:39:06.819 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T02:39:06.826 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T02:39:06.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:39:06.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:39:06.831 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:39:06.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:39:06.868 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T02:39:06.876 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T02:39:06.876 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:39:06.879 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:39:06.893 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:39:06.895 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:39:06.902 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:39:06.931 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:39:06.938 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:39:06.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:39:06.957 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:39:06.964 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:39:07.014 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:39:07.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:39:07.036 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:39:07.043 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:39:07.092 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:39:07.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:39:07.123 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:39:07.130 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:39:07.178 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:39:07.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:39:07.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-09-15T02:39:07.209 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-09-15T02:39:07.211 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-09-15T02:39:07.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-09-15T02:39:07.275 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T02:39:07.282 INFO:teuthology.task.internal:Starting timer... 2024-09-15T02:39:07.282 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T02:39:07.292 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T02:39:07.300 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2024-09-15T02:39:07.300 DEBUG:teuthology.task.selinux:Excluding smithi118: OS 'ubuntu' does not support SELinux 2024-09-15T02:39:07.300 DEBUG:teuthology.task.selinux:Excluding smithi152: OS 'ubuntu' does not support SELinux 2024-09-15T02:39:07.300 DEBUG:teuthology.task.selinux:Excluding smithi191: OS 'ubuntu' does not support SELinux 2024-09-15T02:39:07.300 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T02:39:07.300 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-15T02:39:07.300 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T02:39:07.300 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T02:39:07.310 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T02:39:07.322 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T02:39:07.323 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi118.front.sepia.ceph.com,smithi152.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T02:44:50.885 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-09-15T02:44:50.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-15T02:44:50.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:44:51.003 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-15T02:44:51.130 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-15T02:44:51.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:44:51.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:44:51.228 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-15T02:44:51.350 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-15T02:44:51.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:44:51.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:44:51.436 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-15T02:44:51.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-15T02:44:51.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-15T02:44:51.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:44:51.650 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-15T02:44:51.775 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-15T02:44:51.776 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T02:44:51.787 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T02:44:51.787 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:44:51.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:44:51.790 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:44:51.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:44:51.792 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:44:51.793 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:44:51.795 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:44:51.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:44:51.824 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T02:44:51.824 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Command line: ntpd -gq 2024-09-15T02:44:51.825 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: proto: precision = 0.120 usec (-23) 2024-09-15T02:44:51.825 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.826 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.827 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listen normally on 3 ens1f0 172.21.15.6:123 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listen normally on 4 lo [::1]:123 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%4]:123 2024-09-15T02:44:51.828 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:51 ntpd[13665]: Listening on routing socket on fd #22 for interface updates 2024-09-15T02:44:51.829 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T02:44:51.829 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Command line: ntpd -gq 2024-09-15T02:44:51.829 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: proto: precision = 0.100 usec (-23) 2024-09-15T02:44:51.830 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T02:44:51.830 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Command line: ntpd -gq 2024-09-15T02:44:51.830 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: proto: precision = 0.108 usec (-23) 2024-09-15T02:44:51.830 INFO:teuthology.orchestra.run.smithi152.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.830 INFO:teuthology.orchestra.run.smithi118.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.831 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.831 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.832 INFO:teuthology.orchestra.run.smithi152.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.832 INFO:teuthology.orchestra.run.smithi118.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.833 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listen normally on 3 enp3s0f1 172.21.15.152:123 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listen normally on 4 lo [::1]:123 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6de3%5]:123 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:51 ntpd[13745]: Listening on routing socket on fd #22 for interface updates 2024-09-15T02:44:51.834 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listen normally on 3 enp3s0f1 172.21.15.118:123 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listen normally on 4 lo [::1]:123 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d725%5]:123 2024-09-15T02:44:51.835 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:51 ntpd[13755]: Listening on routing socket on fd #22 for interface updates 2024-09-15T02:44:51.836 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T02:44:51.836 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Command line: ntpd -gq 2024-09-15T02:44:51.836 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: proto: precision = 0.116 usec (-23) 2024-09-15T02:44:51.836 INFO:teuthology.orchestra.run.smithi191.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.837 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:44:51.837 INFO:teuthology.orchestra.run.smithi191.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listen normally on 3 enp3s0f1 172.21.15.191:123 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listen normally on 4 lo [::1]:123 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7211%5]:123 2024-09-15T02:44:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:51 ntpd[13718]: Listening on routing socket on fd #22 for interface updates 2024-09-15T02:44:52.825 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.825 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.825 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:52 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.829 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.829 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:52 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:52.829 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:52 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:52 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:52 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:52 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:52 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:52 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:52.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.832 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.832 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.832 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.832 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:52.832 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:52.833 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:52.833 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:52.833 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:52 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:53.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:53 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:53.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:53 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:53.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:53 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:53.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:53 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:53.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:53 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:53.828 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:53 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:53.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:53 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:53.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:53 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:53.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:53 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:53.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:53 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:53.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:53 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:53.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:53 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:54 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.830 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:54 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:54 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:54.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:54 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:55.824 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:55 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:55.824 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:55 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:55.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:55 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:55.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:55 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:55.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:55 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:55.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:55 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:55.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:55 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:55.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:55 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:55 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:55 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:55 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:55 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:55 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:55.832 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:55 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:56 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:56 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.831 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.833 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.833 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:56 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.834 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:56.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:56.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:56.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:56.835 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:56 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:57.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:57 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:57.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:57 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:57.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:57 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:57.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:57 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:57.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:57 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:57.827 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:57 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:57.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:57 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:57.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:57 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:57.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:57 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:57.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:57 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:57.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:57 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:57.831 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:57 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.825 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.825 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.825 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:15 Sep 02:44:58 ntpd[13665]: ntpd: time slew +0.002111 s 2024-09-15T02:44:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew +0.002111s 2024-09-15T02:44:58.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T02:44:58.827 INFO:teuthology.orchestra.run.smithi006.stderr:15 Sep 02:44:58 ntpd[13665]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T02:44:58.829 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.831 INFO:teuthology.orchestra.run.smithi118.stdout:15 Sep 02:44:58 ntpd[13755]: ntpd: time slew +0.000329 s 2024-09-15T02:44:58.831 INFO:teuthology.orchestra.run.smithi118.stdout:ntpd: time slew +0.000329s 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:15 Sep 02:44:58 ntpd[13755]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.832 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.836 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.836 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:58.836 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.836 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.836 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.837 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:58.837 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.837 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.837 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.837 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:58.837 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.838 INFO:teuthology.orchestra.run.smithi152.stdout:15 Sep 02:44:58 ntpd[13745]: ntpd: time slew +0.001816 s 2024-09-15T02:44:58.838 INFO:teuthology.orchestra.run.smithi152.stdout:ntpd: time slew +0.001816s 2024-09-15T02:44:58.839 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T02:44:58.839 INFO:teuthology.orchestra.run.smithi152.stderr:15 Sep 02:44:58 ntpd[13745]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:44:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:44:58.841 INFO:teuthology.orchestra.run.smithi191.stdout:15 Sep 02:44:58 ntpd[13718]: ntpd: time slew +0.000972 s 2024-09-15T02:44:58.841 INFO:teuthology.orchestra.run.smithi191.stdout:ntpd: time slew +0.000972s 2024-09-15T02:44:58.841 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T02:44:58.841 INFO:teuthology.orchestra.run.smithi191.stderr:15 Sep 02:44:58 ntpd[13718]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T02:44:58.890 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:44:58.890 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-09-15T02:44:58.890 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.890 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.891 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.891 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:44:58.903 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2024-09-15T02:44:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:44:58.905 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================== 2024-09-15T02:44:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.942 INFO:teuthology.orchestra.run.smithi152.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:44:58.942 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================== 2024-09-15T02:44:58.942 INFO:teuthology.orchestra.run.smithi152.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.942 INFO:teuthology.orchestra.run.smithi152.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.942 INFO:teuthology.orchestra.run.smithi152.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.943 INFO:teuthology.orchestra.run.smithi152.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:44:58.943 INFO:teuthology.run_tasks:Running task install... 2024-09-15T02:44:58.952 DEBUG:teuthology.task.install:project ceph 2024-09-15T02:44:58.952 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:44:58.952 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T02:44:58.952 INFO:teuthology.task.install:Using flavor: default 2024-09-15T02:44:58.957 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T02:44:58.957 INFO:teuthology.task.install:extra packages: [] 2024-09-15T02:44:58.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-key list | grep Ceph 2024-09-15T02:44:58.958 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-key list | grep Ceph 2024-09-15T02:44:58.959 DEBUG:teuthology.orchestra.run.smithi152:> sudo apt-key list | grep Ceph 2024-09-15T02:44:58.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-key list | grep Ceph 2024-09-15T02:44:59.060 INFO:teuthology.orchestra.run.smithi118.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T02:44:59.060 INFO:teuthology.orchestra.run.smithi191.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T02:44:59.062 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T02:44:59.088 INFO:teuthology.orchestra.run.smithi152.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T02:44:59.121 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T02:44:59.122 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T02:44:59.122 INFO:teuthology.orchestra.run.smithi191.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T02:44:59.122 INFO:teuthology.orchestra.run.smithi191.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T02:44:59.122 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T02:44:59.122 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:44:59.125 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T02:44:59.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:44:59.131 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T02:44:59.131 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T02:44:59.131 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T02:44:59.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:44:59.174 INFO:teuthology.orchestra.run.smithi152.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T02:44:59.174 INFO:teuthology.orchestra.run.smithi152.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T02:44:59.175 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T02:44:59.175 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:44:59.292 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T02:44:59.292 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:44:59.297 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T02:44:59.297 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:44:59.301 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T02:44:59.301 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:44:59.339 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T02:44:59.339 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:44:59.410 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:44:59.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T02:44:59.413 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:44:59.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T02:44:59.418 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:44:59.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T02:44:59.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-09-15T02:44:59.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-15T02:44:59.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-get update 2024-09-15T02:44:59.451 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:44:59.451 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T02:44:59.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo apt-get update 2024-09-15T02:44:59.571 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:44:59.572 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:44:59.577 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:44:59.594 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:44:59.624 INFO:teuthology.orchestra.run.smithi191.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T02:44:59.633 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T02:44:59.645 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T02:44:59.656 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T02:44:59.656 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:44:59.666 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T02:44:59.669 INFO:teuthology.orchestra.run.smithi191.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T02:44:59.678 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T02:44:59.688 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T02:44:59.690 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T02:44:59.707 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:44:59.707 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:44:59.708 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T02:44:59.708 INFO:teuthology.orchestra.run.smithi152.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T02:44:59.711 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T02:44:59.724 INFO:teuthology.orchestra.run.smithi191.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T02:44:59.729 INFO:teuthology.orchestra.run.smithi152.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T02:44:59.740 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T02:44:59.741 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:44:59.741 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:44:59.742 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T02:44:59.752 INFO:teuthology.orchestra.run.smithi152.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T02:44:59.772 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:44:59.774 INFO:teuthology.orchestra.run.smithi152.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T02:44:59.775 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T02:44:59.777 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:44:59.808 INFO:teuthology.orchestra.run.smithi152.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T02:44:59.811 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:44:59.854 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:44:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:44:59.897 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:44:59.935 INFO:teuthology.orchestra.run.smithi152.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:45:00.220 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 28.5 kB in 1s (45.7 kB/s) 2024-09-15T02:45:00.304 INFO:teuthology.orchestra.run.smithi152.stdout:Fetched 28.5 kB in 1s (45.2 kB/s) 2024-09-15T02:45:00.379 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 28.5 kB in 1s (36.6 kB/s) 2024-09-15T02:45:00.407 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 28.5 kB in 1s (36.7 kB/s) 2024-09-15T02:45:01.379 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2024-09-15T02:45:01.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T02:45:01.457 INFO:teuthology.orchestra.run.smithi152.stdout:Reading package lists... 2024-09-15T02:45:01.475 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2024-09-15T02:45:01.482 DEBUG:teuthology.orchestra.run.smithi152:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T02:45:01.551 INFO:teuthology.orchestra.run.smithi152.stdout:Reading package lists... 2024-09-15T02:45:01.558 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-15T02:45:01.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T02:45:01.585 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-15T02:45:01.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T02:45:01.658 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-15T02:45:01.669 INFO:teuthology.orchestra.run.smithi191.stdout:Building dependency tree... 2024-09-15T02:45:01.670 INFO:teuthology.orchestra.run.smithi191.stdout:Reading state information... 2024-09-15T02:45:01.684 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-15T02:45:01.766 INFO:teuthology.orchestra.run.smithi152.stdout:Building dependency tree... 2024-09-15T02:45:01.766 INFO:teuthology.orchestra.run.smithi152.stdout:Reading state information... 2024-09-15T02:45:01.862 INFO:teuthology.orchestra.run.smithi191.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout:The following additional packages will be installed: 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T02:45:01.864 INFO:teuthology.orchestra.run.smithi191.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T02:45:01.865 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T02:45:01.865 INFO:teuthology.orchestra.run.smithi191.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T02:45:01.865 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T02:45:01.866 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T02:45:01.867 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T02:45:01.867 INFO:teuthology.orchestra.run.smithi191.stdout: socat xmlstarlet 2024-09-15T02:45:01.867 INFO:teuthology.orchestra.run.smithi191.stdout:Suggested packages: 2024-09-15T02:45:01.867 INFO:teuthology.orchestra.run.smithi191.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T02:45:01.867 INFO:teuthology.orchestra.run.smithi191.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout:Recommended packages: 2024-09-15T02:45:01.868 INFO:teuthology.orchestra.run.smithi191.stdout: btrfs-tools 2024-09-15T02:45:01.876 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-09-15T02:45:01.877 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-09-15T02:45:01.908 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-15T02:45:01.909 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-15T02:45:01.964 INFO:teuthology.orchestra.run.smithi152.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T02:45:01.965 INFO:teuthology.orchestra.run.smithi152.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T02:45:01.965 INFO:teuthology.orchestra.run.smithi152.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T02:45:01.965 INFO:teuthology.orchestra.run.smithi152.stdout:The following additional packages will be installed: 2024-09-15T02:45:01.965 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T02:45:01.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T02:45:01.968 INFO:teuthology.orchestra.run.smithi152.stdout: socat xmlstarlet 2024-09-15T02:45:01.969 INFO:teuthology.orchestra.run.smithi152.stdout:Suggested packages: 2024-09-15T02:45:01.969 INFO:teuthology.orchestra.run.smithi152.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T02:45:01.969 INFO:teuthology.orchestra.run.smithi152.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T02:45:01.969 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T02:45:01.969 INFO:teuthology.orchestra.run.smithi152.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout:Recommended packages: 2024-09-15T02:45:01.970 INFO:teuthology.orchestra.run.smithi152.stdout: btrfs-tools 2024-09-15T02:45:01.989 INFO:teuthology.orchestra.run.smithi191.stdout:The following NEW packages will be installed: 2024-09-15T02:45:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T02:45:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T02:45:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T02:45:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T02:45:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T02:45:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T02:45:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T02:45:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T02:45:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T02:45:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T02:45:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T02:45:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T02:45:01.992 INFO:teuthology.orchestra.run.smithi191.stdout:The following packages will be upgraded: 2024-09-15T02:45:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 librbd1 2024-09-15T02:45:02.055 INFO:teuthology.orchestra.run.smithi191.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T02:45:02.055 INFO:teuthology.orchestra.run.smithi191.stdout:Need to get 201 MB of archives. 2024-09-15T02:45:02.056 INFO:teuthology.orchestra.run.smithi191.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T02:45:02.056 INFO:teuthology.orchestra.run.smithi191.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T02:45:02.080 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T02:45:02.081 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T02:45:02.081 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T02:45:02.081 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2024-09-15T02:45:02.081 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T02:45:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T02:45:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T02:45:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: socat xmlstarlet 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T02:45:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2024-09-15T02:45:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: btrfs-tools 2024-09-15T02:45:02.091 INFO:teuthology.orchestra.run.smithi152.stdout:The following NEW packages will be installed: 2024-09-15T02:45:02.091 INFO:teuthology.orchestra.run.smithi152.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T02:45:02.091 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T02:45:02.091 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T02:45:02.091 INFO:teuthology.orchestra.run.smithi152.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T02:45:02.092 INFO:teuthology.orchestra.run.smithi152.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T02:45:02.092 INFO:teuthology.orchestra.run.smithi152.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T02:45:02.092 INFO:teuthology.orchestra.run.smithi152.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T02:45:02.093 INFO:teuthology.orchestra.run.smithi152.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T02:45:02.093 INFO:teuthology.orchestra.run.smithi152.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T02:45:02.093 INFO:teuthology.orchestra.run.smithi152.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T02:45:02.093 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T02:45:02.093 INFO:teuthology.orchestra.run.smithi152.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T02:45:02.094 INFO:teuthology.orchestra.run.smithi152.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T02:45:02.095 INFO:teuthology.orchestra.run.smithi152.stdout:The following packages will be upgraded: 2024-09-15T02:45:02.095 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 librbd1 2024-09-15T02:45:02.122 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T02:45:02.126 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T02:45:02.127 INFO:teuthology.orchestra.run.smithi006.stdout: socat xmlstarlet 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T02:45:02.128 INFO:teuthology.orchestra.run.smithi006.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T02:45:02.129 INFO:teuthology.orchestra.run.smithi006.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T02:45:02.129 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-09-15T02:45:02.129 INFO:teuthology.orchestra.run.smithi006.stdout: btrfs-tools 2024-09-15T02:45:02.137 INFO:teuthology.orchestra.run.smithi191.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T02:45:02.138 INFO:teuthology.orchestra.run.smithi191.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T02:45:02.139 INFO:teuthology.orchestra.run.smithi191.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T02:45:02.148 INFO:teuthology.orchestra.run.smithi191.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T02:45:02.154 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T02:45:02.159 INFO:teuthology.orchestra.run.smithi152.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T02:45:02.159 INFO:teuthology.orchestra.run.smithi152.stdout:Need to get 201 MB of archives. 2024-09-15T02:45:02.159 INFO:teuthology.orchestra.run.smithi152.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T02:45:02.159 INFO:teuthology.orchestra.run.smithi152.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T02:45:02.177 INFO:teuthology.orchestra.run.smithi191.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T02:45:02.178 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T02:45:02.183 INFO:teuthology.orchestra.run.smithi191.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T02:45:02.203 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2024-09-15T02:45:02.203 INFO:teuthology.orchestra.run.smithi118.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T02:45:02.203 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T02:45:02.203 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T02:45:02.204 INFO:teuthology.orchestra.run.smithi118.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T02:45:02.204 INFO:teuthology.orchestra.run.smithi118.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T02:45:02.204 INFO:teuthology.orchestra.run.smithi118.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T02:45:02.204 INFO:teuthology.orchestra.run.smithi118.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T02:45:02.204 INFO:teuthology.orchestra.run.smithi118.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T02:45:02.205 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T02:45:02.206 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T02:45:02.206 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T02:45:02.206 INFO:teuthology.orchestra.run.smithi118.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T02:45:02.206 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages will be upgraded: 2024-09-15T02:45:02.206 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 librbd1 2024-09-15T02:45:02.215 INFO:teuthology.orchestra.run.smithi191.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T02:45:02.216 INFO:teuthology.orchestra.run.smithi191.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T02:45:02.222 INFO:teuthology.orchestra.run.smithi191.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T02:45:02.224 INFO:teuthology.orchestra.run.smithi191.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T02:45:02.224 INFO:teuthology.orchestra.run.smithi191.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T02:45:02.227 INFO:teuthology.orchestra.run.smithi191.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T02:45:02.227 INFO:teuthology.orchestra.run.smithi191.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T02:45:02.227 INFO:teuthology.orchestra.run.smithi191.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T02:45:02.228 INFO:teuthology.orchestra.run.smithi191.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T02:45:02.232 INFO:teuthology.orchestra.run.smithi191.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T02:45:02.242 INFO:teuthology.orchestra.run.smithi152.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T02:45:02.242 INFO:teuthology.orchestra.run.smithi152.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T02:45:02.243 INFO:teuthology.orchestra.run.smithi152.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T02:45:02.247 INFO:teuthology.orchestra.run.smithi191.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T02:45:02.247 INFO:teuthology.orchestra.run.smithi191.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T02:45:02.247 INFO:teuthology.orchestra.run.smithi191.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T02:45:02.248 INFO:teuthology.orchestra.run.smithi191.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T02:45:02.250 INFO:teuthology.orchestra.run.smithi006.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T02:45:02.251 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T02:45:02.252 INFO:teuthology.orchestra.run.smithi006.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T02:45:02.252 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T02:45:02.252 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T02:45:02.252 INFO:teuthology.orchestra.run.smithi006.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T02:45:02.252 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2024-09-15T02:45:02.253 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 librbd1 2024-09-15T02:45:02.253 INFO:teuthology.orchestra.run.smithi191.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T02:45:02.253 INFO:teuthology.orchestra.run.smithi191.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T02:45:02.253 INFO:teuthology.orchestra.run.smithi191.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T02:45:02.253 INFO:teuthology.orchestra.run.smithi191.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T02:45:02.253 INFO:teuthology.orchestra.run.smithi191.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T02:45:02.254 INFO:teuthology.orchestra.run.smithi152.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T02:45:02.255 INFO:teuthology.orchestra.run.smithi152.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T02:45:02.265 INFO:teuthology.orchestra.run.smithi191.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T02:45:02.266 INFO:teuthology.orchestra.run.smithi191.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T02:45:02.267 INFO:teuthology.orchestra.run.smithi191.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T02:45:02.273 INFO:teuthology.orchestra.run.smithi118.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T02:45:02.273 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 201 MB of archives. 2024-09-15T02:45:02.273 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T02:45:02.273 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T02:45:02.282 INFO:teuthology.orchestra.run.smithi191.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T02:45:02.283 INFO:teuthology.orchestra.run.smithi152.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T02:45:02.283 INFO:teuthology.orchestra.run.smithi191.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T02:45:02.283 INFO:teuthology.orchestra.run.smithi191.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T02:45:02.283 INFO:teuthology.orchestra.run.smithi191.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T02:45:02.284 INFO:teuthology.orchestra.run.smithi152.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T02:45:02.284 INFO:teuthology.orchestra.run.smithi191.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T02:45:02.287 INFO:teuthology.orchestra.run.smithi191.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T02:45:02.289 INFO:teuthology.orchestra.run.smithi191.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T02:45:02.289 INFO:teuthology.orchestra.run.smithi152.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T02:45:02.299 INFO:teuthology.orchestra.run.smithi191.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T02:45:02.301 INFO:teuthology.orchestra.run.smithi191.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T02:45:02.302 INFO:teuthology.orchestra.run.smithi191.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T02:45:02.308 INFO:teuthology.orchestra.run.smithi191.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi006.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 201 MB of archives. 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi191.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi191.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T02:45:02.317 INFO:teuthology.orchestra.run.smithi191.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T02:45:02.320 INFO:teuthology.orchestra.run.smithi152.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T02:45:02.322 INFO:teuthology.orchestra.run.smithi152.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T02:45:02.328 INFO:teuthology.orchestra.run.smithi152.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T02:45:02.330 INFO:teuthology.orchestra.run.smithi152.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T02:45:02.330 INFO:teuthology.orchestra.run.smithi152.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T02:45:02.332 INFO:teuthology.orchestra.run.smithi152.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T02:45:02.333 INFO:teuthology.orchestra.run.smithi152.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T02:45:02.333 INFO:teuthology.orchestra.run.smithi152.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T02:45:02.333 INFO:teuthology.orchestra.run.smithi152.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T02:45:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T02:45:02.352 INFO:teuthology.orchestra.run.smithi152.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T02:45:02.353 INFO:teuthology.orchestra.run.smithi152.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T02:45:02.354 INFO:teuthology.orchestra.run.smithi152.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T02:45:02.354 INFO:teuthology.orchestra.run.smithi152.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T02:45:02.356 INFO:teuthology.orchestra.run.smithi152.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T02:45:02.357 INFO:teuthology.orchestra.run.smithi152.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T02:45:02.358 INFO:teuthology.orchestra.run.smithi152.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T02:45:02.358 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T02:45:02.360 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T02:45:02.360 INFO:teuthology.orchestra.run.smithi152.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T02:45:02.361 INFO:teuthology.orchestra.run.smithi152.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T02:45:02.364 INFO:teuthology.orchestra.run.smithi191.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T02:45:02.369 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T02:45:02.371 INFO:teuthology.orchestra.run.smithi152.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T02:45:02.372 INFO:teuthology.orchestra.run.smithi152.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T02:45:02.373 INFO:teuthology.orchestra.run.smithi152.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T02:45:02.376 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T02:45:02.376 INFO:teuthology.orchestra.run.smithi191.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T02:45:02.388 INFO:teuthology.orchestra.run.smithi152.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T02:45:02.388 INFO:teuthology.orchestra.run.smithi152.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T02:45:02.389 INFO:teuthology.orchestra.run.smithi152.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T02:45:02.390 INFO:teuthology.orchestra.run.smithi152.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T02:45:02.390 INFO:teuthology.orchestra.run.smithi191.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T02:45:02.391 INFO:teuthology.orchestra.run.smithi191.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T02:45:02.391 INFO:teuthology.orchestra.run.smithi152.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T02:45:02.394 INFO:teuthology.orchestra.run.smithi191.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T02:45:02.394 INFO:teuthology.orchestra.run.smithi152.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T02:45:02.395 INFO:teuthology.orchestra.run.smithi191.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T02:45:02.396 INFO:teuthology.orchestra.run.smithi152.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T02:45:02.399 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T02:45:02.400 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T02:45:02.400 INFO:teuthology.orchestra.run.smithi191.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T02:45:02.400 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T02:45:02.401 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T02:45:02.401 INFO:teuthology.orchestra.run.smithi191.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T02:45:02.405 INFO:teuthology.orchestra.run.smithi152.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T02:45:02.406 INFO:teuthology.orchestra.run.smithi152.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T02:45:02.407 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T02:45:02.407 INFO:teuthology.orchestra.run.smithi152.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T02:45:02.409 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T02:45:02.415 INFO:teuthology.orchestra.run.smithi191.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T02:45:02.416 INFO:teuthology.orchestra.run.smithi191.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T02:45:02.416 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T02:45:02.422 INFO:teuthology.orchestra.run.smithi152.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T02:45:02.422 INFO:teuthology.orchestra.run.smithi152.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T02:45:02.424 INFO:teuthology.orchestra.run.smithi152.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T02:45:02.428 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T02:45:02.428 INFO:teuthology.orchestra.run.smithi191.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T02:45:02.428 INFO:teuthology.orchestra.run.smithi191.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T02:45:02.428 INFO:teuthology.orchestra.run.smithi191.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T02:45:02.440 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T02:45:02.441 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T02:45:02.449 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T02:45:02.451 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T02:45:02.454 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T02:45:02.462 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T02:45:02.464 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T02:45:02.464 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T02:45:02.466 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T02:45:02.467 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T02:45:02.467 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T02:45:02.467 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T02:45:02.470 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T02:45:02.481 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T02:45:02.483 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T02:45:02.484 INFO:teuthology.orchestra.run.smithi191.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T02:45:02.485 INFO:teuthology.orchestra.run.smithi191.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T02:45:02.486 INFO:teuthology.orchestra.run.smithi191.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T02:45:02.487 INFO:teuthology.orchestra.run.smithi191.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T02:45:02.487 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T02:45:02.487 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T02:45:02.488 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T02:45:02.488 INFO:teuthology.orchestra.run.smithi191.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T02:45:02.489 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T02:45:02.489 INFO:teuthology.orchestra.run.smithi191.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T02:45:02.489 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T02:45:02.490 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T02:45:02.490 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T02:45:02.491 INFO:teuthology.orchestra.run.smithi191.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T02:45:02.491 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T02:45:02.491 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T02:45:02.492 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T02:45:02.492 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T02:45:02.493 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T02:45:02.494 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T02:45:02.494 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T02:45:02.495 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T02:45:02.495 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T02:45:02.496 INFO:teuthology.orchestra.run.smithi191.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T02:45:02.498 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T02:45:02.501 INFO:teuthology.orchestra.run.smithi191.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T02:45:02.504 INFO:teuthology.orchestra.run.smithi152.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T02:45:02.505 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T02:45:02.511 INFO:teuthology.orchestra.run.smithi191.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T02:45:02.512 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T02:45:02.512 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T02:45:02.512 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T02:45:02.513 INFO:teuthology.orchestra.run.smithi191.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T02:45:02.514 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T02:45:02.514 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T02:45:02.515 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T02:45:02.515 INFO:teuthology.orchestra.run.smithi191.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T02:45:02.515 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T02:45:02.516 INFO:teuthology.orchestra.run.smithi191.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T02:45:02.516 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T02:45:02.517 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T02:45:02.520 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T02:45:02.520 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T02:45:02.520 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T02:45:02.521 INFO:teuthology.orchestra.run.smithi191.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi191.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi152.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi152.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T02:45:02.522 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T02:45:02.523 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T02:45:02.523 INFO:teuthology.orchestra.run.smithi191.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T02:45:02.525 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T02:45:02.527 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T02:45:02.532 INFO:teuthology.orchestra.run.smithi152.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T02:45:02.532 INFO:teuthology.orchestra.run.smithi152.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T02:45:02.533 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T02:45:02.533 INFO:teuthology.orchestra.run.smithi152.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T02:45:02.534 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T02:45:02.540 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T02:45:02.541 INFO:teuthology.orchestra.run.smithi191.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T02:45:02.541 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T02:45:02.543 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T02:45:02.544 INFO:teuthology.orchestra.run.smithi191.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T02:45:02.544 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T02:45:02.545 INFO:teuthology.orchestra.run.smithi118.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T02:45:02.546 INFO:teuthology.orchestra.run.smithi118.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T02:45:02.550 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T02:45:02.557 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T02:45:02.557 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T02:45:02.558 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T02:45:02.559 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T02:45:02.560 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T02:45:02.562 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T02:45:02.564 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T02:45:02.566 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T02:45:02.568 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T02:45:02.579 INFO:teuthology.orchestra.run.smithi152.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T02:45:02.580 INFO:teuthology.orchestra.run.smithi152.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T02:45:02.581 INFO:teuthology.orchestra.run.smithi152.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T02:45:02.581 INFO:teuthology.orchestra.run.smithi152.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T02:45:02.582 INFO:teuthology.orchestra.run.smithi152.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T02:45:02.583 INFO:teuthology.orchestra.run.smithi152.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T02:45:02.583 INFO:teuthology.orchestra.run.smithi191.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T02:45:02.584 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T02:45:02.584 INFO:teuthology.orchestra.run.smithi191.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T02:45:02.584 INFO:teuthology.orchestra.run.smithi191.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T02:45:02.584 INFO:teuthology.orchestra.run.smithi191.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T02:45:02.585 INFO:teuthology.orchestra.run.smithi191.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T02:45:02.585 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T02:45:02.585 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T02:45:02.586 INFO:teuthology.orchestra.run.smithi191.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T02:45:02.586 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T02:45:02.597 INFO:teuthology.orchestra.run.smithi152.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T02:45:02.597 INFO:teuthology.orchestra.run.smithi191.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T02:45:02.597 INFO:teuthology.orchestra.run.smithi191.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T02:45:02.600 INFO:teuthology.orchestra.run.smithi152.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T02:45:02.602 INFO:teuthology.orchestra.run.smithi152.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T02:45:02.604 INFO:teuthology.orchestra.run.smithi152.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T02:45:02.614 INFO:teuthology.orchestra.run.smithi152.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T02:45:02.617 INFO:teuthology.orchestra.run.smithi152.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T02:45:02.621 INFO:teuthology.orchestra.run.smithi152.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T02:45:02.625 INFO:teuthology.orchestra.run.smithi118.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T02:45:02.638 INFO:teuthology.orchestra.run.smithi152.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T02:45:02.640 INFO:teuthology.orchestra.run.smithi118.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T02:45:02.640 INFO:teuthology.orchestra.run.smithi152.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T02:45:02.641 INFO:teuthology.orchestra.run.smithi118.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T02:45:02.642 INFO:teuthology.orchestra.run.smithi191.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T02:45:02.644 INFO:teuthology.orchestra.run.smithi191.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T02:45:02.645 INFO:teuthology.orchestra.run.smithi191.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T02:45:02.646 INFO:teuthology.orchestra.run.smithi191.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T02:45:02.646 INFO:teuthology.orchestra.run.smithi191.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T02:45:02.648 INFO:teuthology.orchestra.run.smithi191.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T02:45:02.648 INFO:teuthology.orchestra.run.smithi191.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T02:45:02.654 INFO:teuthology.orchestra.run.smithi118.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T02:45:02.669 INFO:teuthology.orchestra.run.smithi006.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T02:45:02.731 INFO:teuthology.orchestra.run.smithi006.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T02:45:02.731 INFO:teuthology.orchestra.run.smithi006.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T02:45:02.731 INFO:teuthology.orchestra.run.smithi006.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T02:45:02.731 INFO:teuthology.orchestra.run.smithi006.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T02:45:02.732 INFO:teuthology.orchestra.run.smithi118.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T02:45:02.732 INFO:teuthology.orchestra.run.smithi118.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T02:45:02.732 INFO:teuthology.orchestra.run.smithi118.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T02:45:02.732 INFO:teuthology.orchestra.run.smithi118.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T02:45:02.733 INFO:teuthology.orchestra.run.smithi118.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T02:45:02.733 INFO:teuthology.orchestra.run.smithi118.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T02:45:02.733 INFO:teuthology.orchestra.run.smithi118.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T02:45:02.733 INFO:teuthology.orchestra.run.smithi118.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T02:45:02.733 INFO:teuthology.orchestra.run.smithi152.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T02:45:02.734 INFO:teuthology.orchestra.run.smithi152.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T02:45:02.735 INFO:teuthology.orchestra.run.smithi152.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T02:45:02.736 INFO:teuthology.orchestra.run.smithi152.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T02:45:02.736 INFO:teuthology.orchestra.run.smithi152.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T02:45:02.739 INFO:teuthology.orchestra.run.smithi118.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T02:45:02.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T02:45:02.747 INFO:teuthology.orchestra.run.smithi118.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T02:45:02.749 INFO:teuthology.orchestra.run.smithi118.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T02:45:02.750 INFO:teuthology.orchestra.run.smithi118.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T02:45:02.753 INFO:teuthology.orchestra.run.smithi118.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T02:45:02.759 INFO:teuthology.orchestra.run.smithi118.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T02:45:02.761 INFO:teuthology.orchestra.run.smithi006.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T02:45:02.763 INFO:teuthology.orchestra.run.smithi006.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T02:45:02.764 INFO:teuthology.orchestra.run.smithi006.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T02:45:02.765 INFO:teuthology.orchestra.run.smithi006.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T02:45:02.765 INFO:teuthology.orchestra.run.smithi006.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T02:45:02.767 INFO:teuthology.orchestra.run.smithi006.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T02:45:02.768 INFO:teuthology.orchestra.run.smithi006.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T02:45:02.775 INFO:teuthology.orchestra.run.smithi118.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T02:45:02.778 INFO:teuthology.orchestra.run.smithi118.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T02:45:02.779 INFO:teuthology.orchestra.run.smithi006.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T02:45:02.782 INFO:teuthology.orchestra.run.smithi006.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T02:45:02.783 INFO:teuthology.orchestra.run.smithi006.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T02:45:02.796 INFO:teuthology.orchestra.run.smithi006.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T02:45:02.797 INFO:teuthology.orchestra.run.smithi006.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T02:45:02.801 INFO:teuthology.orchestra.run.smithi006.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T02:45:02.805 INFO:teuthology.orchestra.run.smithi006.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T02:45:02.815 INFO:teuthology.orchestra.run.smithi118.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T02:45:02.816 INFO:teuthology.orchestra.run.smithi118.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T02:45:02.816 INFO:teuthology.orchestra.run.smithi118.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T02:45:02.817 INFO:teuthology.orchestra.run.smithi118.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T02:45:02.817 INFO:teuthology.orchestra.run.smithi118.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T02:45:02.818 INFO:teuthology.orchestra.run.smithi118.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T02:45:02.823 INFO:teuthology.orchestra.run.smithi006.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T02:45:02.825 INFO:teuthology.orchestra.run.smithi006.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T02:45:02.830 INFO:teuthology.orchestra.run.smithi118.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T02:45:02.830 INFO:teuthology.orchestra.run.smithi118.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T02:45:02.830 INFO:teuthology.orchestra.run.smithi118.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T02:45:02.836 INFO:teuthology.orchestra.run.smithi118.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T02:45:02.854 INFO:teuthology.orchestra.run.smithi118.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T02:45:02.854 INFO:teuthology.orchestra.run.smithi118.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T02:45:02.855 INFO:teuthology.orchestra.run.smithi118.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T02:45:02.857 INFO:teuthology.orchestra.run.smithi118.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T02:45:02.858 INFO:teuthology.orchestra.run.smithi118.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T02:45:02.869 INFO:teuthology.orchestra.run.smithi006.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T02:45:02.870 INFO:teuthology.orchestra.run.smithi006.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T02:45:02.870 INFO:teuthology.orchestra.run.smithi006.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T02:45:02.871 INFO:teuthology.orchestra.run.smithi006.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T02:45:02.871 INFO:teuthology.orchestra.run.smithi006.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T02:45:02.873 INFO:teuthology.orchestra.run.smithi006.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T02:45:02.884 INFO:teuthology.orchestra.run.smithi006.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T02:45:02.885 INFO:teuthology.orchestra.run.smithi006.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T02:45:02.910 INFO:teuthology.orchestra.run.smithi006.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T02:45:02.912 INFO:teuthology.orchestra.run.smithi006.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T02:45:02.914 INFO:teuthology.orchestra.run.smithi006.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T02:45:02.914 INFO:teuthology.orchestra.run.smithi006.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T02:45:02.915 INFO:teuthology.orchestra.run.smithi006.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T02:45:02.917 INFO:teuthology.orchestra.run.smithi006.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T02:45:02.918 INFO:teuthology.orchestra.run.smithi006.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T02:45:03.029 INFO:teuthology.orchestra.run.smithi152.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T02:45:03.095 INFO:teuthology.orchestra.run.smithi152.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T02:45:03.150 INFO:teuthology.orchestra.run.smithi152.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T02:45:03.150 INFO:teuthology.orchestra.run.smithi152.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T02:45:03.154 INFO:teuthology.orchestra.run.smithi152.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T02:45:03.155 INFO:teuthology.orchestra.run.smithi152.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T02:45:03.160 INFO:teuthology.orchestra.run.smithi152.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T02:45:03.182 INFO:teuthology.orchestra.run.smithi191.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T02:45:03.242 INFO:teuthology.orchestra.run.smithi191.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T02:45:03.244 INFO:teuthology.orchestra.run.smithi191.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T02:45:03.247 INFO:teuthology.orchestra.run.smithi191.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T02:45:03.283 INFO:teuthology.orchestra.run.smithi191.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T02:45:03.395 INFO:teuthology.orchestra.run.smithi191.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T02:45:03.485 INFO:teuthology.orchestra.run.smithi152.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T02:45:03.517 INFO:teuthology.orchestra.run.smithi152.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T02:45:03.540 INFO:teuthology.orchestra.run.smithi152.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T02:45:03.748 INFO:teuthology.orchestra.run.smithi118.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T02:45:04.110 INFO:teuthology.orchestra.run.smithi191.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T02:45:04.110 INFO:teuthology.orchestra.run.smithi191.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T02:45:04.122 INFO:teuthology.orchestra.run.smithi191.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T02:45:04.140 INFO:teuthology.orchestra.run.smithi191.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T02:45:04.141 INFO:teuthology.orchestra.run.smithi191.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T02:45:04.142 INFO:teuthology.orchestra.run.smithi191.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T02:45:04.235 INFO:teuthology.orchestra.run.smithi118.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T02:45:04.294 INFO:teuthology.orchestra.run.smithi118.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T02:45:04.320 INFO:teuthology.orchestra.run.smithi191.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T02:45:04.344 INFO:teuthology.orchestra.run.smithi118.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T02:45:04.345 INFO:teuthology.orchestra.run.smithi118.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T02:45:04.385 INFO:teuthology.orchestra.run.smithi118.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T02:45:04.387 INFO:teuthology.orchestra.run.smithi118.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T02:45:04.415 INFO:teuthology.orchestra.run.smithi006.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T02:45:04.436 INFO:teuthology.orchestra.run.smithi118.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T02:45:04.543 INFO:teuthology.orchestra.run.smithi191.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T02:45:04.543 INFO:teuthology.orchestra.run.smithi191.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T02:45:04.905 INFO:teuthology.orchestra.run.smithi118.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T02:45:04.907 INFO:teuthology.orchestra.run.smithi118.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T02:45:04.961 INFO:teuthology.orchestra.run.smithi118.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T02:45:05.126 INFO:teuthology.orchestra.run.smithi006.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T02:45:05.260 INFO:teuthology.orchestra.run.smithi006.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T02:45:05.330 INFO:teuthology.orchestra.run.smithi006.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T02:45:05.330 INFO:teuthology.orchestra.run.smithi006.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T02:45:05.337 INFO:teuthology.orchestra.run.smithi006.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T02:45:05.337 INFO:teuthology.orchestra.run.smithi006.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T02:45:05.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T02:45:05.553 INFO:teuthology.orchestra.run.smithi152.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T02:45:05.956 INFO:teuthology.orchestra.run.smithi152.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T02:45:05.970 INFO:teuthology.orchestra.run.smithi152.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T02:45:05.986 INFO:teuthology.orchestra.run.smithi152.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T02:45:06.113 INFO:teuthology.orchestra.run.smithi152.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T02:45:06.173 INFO:teuthology.orchestra.run.smithi006.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T02:45:06.216 INFO:teuthology.orchestra.run.smithi006.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T02:45:06.280 INFO:teuthology.orchestra.run.smithi006.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T02:45:06.499 INFO:teuthology.orchestra.run.smithi152.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T02:45:06.745 INFO:teuthology.orchestra.run.smithi191.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T02:45:06.746 INFO:teuthology.orchestra.run.smithi118.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T02:45:06.806 INFO:teuthology.orchestra.run.smithi191.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T02:45:06.806 INFO:teuthology.orchestra.run.smithi191.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T02:45:06.838 INFO:teuthology.orchestra.run.smithi118.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T02:45:06.841 INFO:teuthology.orchestra.run.smithi118.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T02:45:06.844 INFO:teuthology.orchestra.run.smithi118.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T02:45:06.949 INFO:teuthology.orchestra.run.smithi118.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T02:45:07.220 INFO:teuthology.orchestra.run.smithi118.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T02:45:07.487 INFO:teuthology.orchestra.run.smithi191.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T02:45:08.028 INFO:teuthology.orchestra.run.smithi152.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T02:45:08.029 INFO:teuthology.orchestra.run.smithi152.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T02:45:08.042 INFO:teuthology.orchestra.run.smithi152.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T02:45:08.135 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 201 MB in 5s (36.6 MB/s) 2024-09-15T02:45:08.140 INFO:teuthology.orchestra.run.smithi152.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T02:45:08.142 INFO:teuthology.orchestra.run.smithi152.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T02:45:08.144 INFO:teuthology.orchestra.run.smithi152.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T02:45:08.164 INFO:teuthology.orchestra.run.smithi118.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T02:45:08.165 INFO:teuthology.orchestra.run.smithi118.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T02:45:08.177 INFO:teuthology.orchestra.run.smithi118.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T02:45:08.206 INFO:teuthology.orchestra.run.smithi118.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T02:45:08.207 INFO:teuthology.orchestra.run.smithi118.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T02:45:08.208 INFO:teuthology.orchestra.run.smithi118.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T02:45:08.304 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T02:45:08.361 INFO:teuthology.orchestra.run.smithi191.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T02:45:08.366 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T02:45:08.433 INFO:teuthology.orchestra.run.smithi118.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T02:45:08.469 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:45:08.503 INFO:teuthology.orchestra.run.smithi152.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T02:45:08.700 INFO:teuthology.orchestra.run.smithi118.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T02:45:08.701 INFO:teuthology.orchestra.run.smithi118.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T02:45:08.826 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:08.849 INFO:teuthology.orchestra.run.smithi006.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T02:45:08.901 INFO:teuthology.orchestra.run.smithi152.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T02:45:08.901 INFO:teuthology.orchestra.run.smithi152.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T02:45:08.948 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:09.186 INFO:teuthology.orchestra.run.smithi006.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T02:45:09.225 INFO:teuthology.orchestra.run.smithi006.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T02:45:09.229 INFO:teuthology.orchestra.run.smithi006.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T02:45:09.363 INFO:teuthology.orchestra.run.smithi006.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T02:45:09.685 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:09.729 INFO:teuthology.orchestra.run.smithi006.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T02:45:09.811 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:10.498 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T02:45:10.517 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:10.556 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:10.791 INFO:teuthology.orchestra.run.smithi118.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T02:45:10.797 INFO:teuthology.orchestra.run.smithi118.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T02:45:10.797 INFO:teuthology.orchestra.run.smithi118.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T02:45:10.900 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rados. 2024-09-15T02:45:10.916 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:10.958 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:11.339 INFO:teuthology.orchestra.run.smithi118.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T02:45:11.356 INFO:teuthology.orchestra.run.smithi006.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T02:45:11.357 INFO:teuthology.orchestra.run.smithi006.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T02:45:11.361 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T02:45:11.369 INFO:teuthology.orchestra.run.smithi006.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T02:45:11.381 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:11.419 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:11.442 INFO:teuthology.orchestra.run.smithi006.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T02:45:11.443 INFO:teuthology.orchestra.run.smithi006.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T02:45:11.445 INFO:teuthology.orchestra.run.smithi006.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T02:45:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T02:45:11.697 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:11.709 INFO:teuthology.orchestra.run.smithi006.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T02:45:11.730 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:11.775 INFO:teuthology.orchestra.run.smithi152.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T02:45:11.780 INFO:teuthology.orchestra.run.smithi152.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T02:45:11.781 INFO:teuthology.orchestra.run.smithi152.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T02:45:11.973 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 201 MB in 9s (22.0 MB/s) 2024-09-15T02:45:11.983 INFO:teuthology.orchestra.run.smithi006.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T02:45:11.983 INFO:teuthology.orchestra.run.smithi006.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T02:45:12.074 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T02:45:12.091 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:12.132 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:12.133 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T02:45:12.189 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T02:45:12.193 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T02:45:12.267 INFO:teuthology.orchestra.run.smithi152.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T02:45:12.307 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:45:12.502 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T02:45:12.518 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T02:45:12.561 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T02:45:12.695 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:12.828 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:12.838 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T02:45:12.855 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:12.896 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:12.904 INFO:teuthology.orchestra.run.smithi152.stdout:Fetched 201 MB in 10s (19.8 MB/s) 2024-09-15T02:45:13.101 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T02:45:13.157 INFO:teuthology.orchestra.run.smithi152.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T02:45:13.161 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T02:45:13.283 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:45:13.291 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T02:45:13.308 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T02:45:13.349 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:45:13.558 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:13.662 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:13.685 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T02:45:13.699 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:13.705 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T02:45:13.743 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:45:13.804 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:14.121 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T02:45:14.122 INFO:teuthology.orchestra.run.smithi006.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T02:45:14.127 INFO:teuthology.orchestra.run.smithi006.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T02:45:14.128 INFO:teuthology.orchestra.run.smithi006.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T02:45:14.137 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T02:45:14.179 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:45:14.437 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T02:45:14.452 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:14.486 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:14.500 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T02:45:14.573 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T02:45:14.632 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:45:14.633 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:14.675 INFO:teuthology.orchestra.run.smithi006.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T02:45:14.847 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rados. 2024-09-15T02:45:14.866 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:14.902 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T02:45:14.905 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:14.921 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T02:45:14.971 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:45:15.292 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T02:45:15.308 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:15.355 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 201 MB in 12s (16.2 MB/s) 2024-09-15T02:45:15.369 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T02:45:15.378 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:15.388 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:15.427 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:15.463 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T02:45:15.480 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:15.532 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:15.540 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T02:45:15.597 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T02:45:15.601 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T02:45:15.620 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T02:45:15.636 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:15.678 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:15.705 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:45:15.771 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-rados. 2024-09-15T02:45:15.787 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:15.854 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:16.002 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:16.039 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T02:45:16.057 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:16.083 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T02:45:16.100 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:16.114 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:16.126 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:16.157 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:16.215 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T02:45:16.233 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:16.275 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:16.461 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T02:45:16.478 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T02:45:16.526 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T02:45:16.527 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T02:45:16.546 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:16.569 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T02:45:16.586 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:16.586 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:16.627 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:16.809 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:16.820 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T02:45:16.838 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:16.879 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:16.964 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:45:16.980 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T02:45:16.988 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T02:45:17.000 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:17.007 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T02:45:17.030 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:17.047 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:45:17.256 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T02:45:17.275 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T02:45:17.314 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:45:17.350 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T02:45:17.369 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T02:45:17.399 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T02:45:17.416 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librgw2. 2024-09-15T02:45:17.432 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:17.474 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:17.592 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T02:45:17.610 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:17.693 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T02:45:17.694 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T02:45:17.708 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T02:45:17.709 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:17.710 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:17.751 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:45:17.752 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:18.062 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T02:45:18.079 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T02:45:18.112 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rados. 2024-09-15T02:45:18.112 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:45:18.128 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:18.146 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T02:45:18.165 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T02:45:18.170 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:18.204 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:45:18.260 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T02:45:18.277 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:18.318 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:18.509 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T02:45:18.523 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T02:45:18.565 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T02:45:18.582 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:18.623 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:18.653 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:45:18.671 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liboath0. 2024-09-15T02:45:18.690 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T02:45:18.729 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T02:45:18.732 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T02:45:18.751 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T02:45:18.815 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:45:18.858 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T02:45:18.876 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:18.917 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:19.065 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T02:45:19.069 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T02:45:19.083 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:19.084 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T02:45:19.124 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:19.127 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:45:19.152 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T02:45:19.169 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T02:45:19.210 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:45:19.219 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T02:45:19.238 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:19.269 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:19.547 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T02:45:19.564 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:19.573 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T02:45:19.593 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T02:45:19.613 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T02:45:19.630 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T02:45:19.631 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T02:45:19.641 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:19.696 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:45:19.836 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-common. 2024-09-15T02:45:19.852 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T02:45:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:19.893 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:20.049 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T02:45:20.064 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T02:45:20.098 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:45:20.150 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T02:45:20.170 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:20.209 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:20.261 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T02:45:20.281 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T02:45:20.319 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:45:20.594 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T02:45:20.595 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T02:45:20.613 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:20.614 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:20.639 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T02:45:20.656 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T02:45:20.662 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:20.679 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:20.698 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:45:21.042 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T02:45:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T02:45:21.065 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T02:45:21.083 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T02:45:21.100 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:45:21.123 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:45:21.196 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T02:45:21.212 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:21.254 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:21.503 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T02:45:21.521 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T02:45:21.527 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librgw2. 2024-09-15T02:45:21.543 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:21.561 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:45:21.585 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:21.624 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T02:45:21.640 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:21.707 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:21.831 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T02:45:21.850 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T02:45:21.881 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:45:22.127 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T02:45:22.143 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T02:45:22.185 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:45:22.379 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T02:45:22.382 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-base. 2024-09-15T02:45:22.384 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T02:45:22.398 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:22.398 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:22.402 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:22.437 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:22.453 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:22.476 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:22.596 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package librgw2. 2024-09-15T02:45:22.612 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:22.654 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:22.748 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liboath0. 2024-09-15T02:45:22.764 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T02:45:22.830 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T02:45:22.946 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T02:45:22.965 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:23.036 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:23.151 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T02:45:23.170 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:23.218 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:23.378 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T02:45:23.391 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T02:45:23.395 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T02:45:23.407 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:45:23.437 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:45:23.449 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:45:23.499 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T02:45:23.517 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:23.557 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:23.688 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-common. 2024-09-15T02:45:23.706 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:23.746 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:23.857 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T02:45:23.874 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T02:45:23.877 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T02:45:23.894 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T02:45:23.915 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T02:45:23.935 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:45:23.959 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package liboath0. 2024-09-15T02:45:23.975 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T02:45:24.039 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T02:45:24.276 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T02:45:24.280 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librgw2. 2024-09-15T02:45:24.292 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T02:45:24.298 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:24.334 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T02:45:24.338 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:24.379 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T02:45:24.397 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:24.437 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:24.595 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T02:45:24.612 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:24.653 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:24.965 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-common. 2024-09-15T02:45:24.980 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:25.023 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:25.050 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T02:45:25.065 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T02:45:25.099 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T02:45:25.108 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:45:25.117 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:25.149 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:25.444 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T02:45:25.460 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T02:45:25.502 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liboath0. 2024-09-15T02:45:25.502 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:45:25.521 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T02:45:25.560 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T02:45:25.821 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T02:45:25.838 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:25.871 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:25.872 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T02:45:25.888 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T02:45:25.930 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T02:45:26.283 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T02:45:26.285 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-base. 2024-09-15T02:45:26.300 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T02:45:26.303 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:26.342 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:45:26.360 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-common. 2024-09-15T02:45:26.376 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:26.387 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:26.418 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:26.670 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T02:45:26.686 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T02:45:26.728 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T02:45:27.064 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T02:45:27.080 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T02:45:27.130 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:45:27.274 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T02:45:27.289 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T02:45:27.332 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:45:27.467 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-mako. 2024-09-15T02:45:27.483 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T02:45:27.525 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:45:27.653 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-base. 2024-09-15T02:45:27.670 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:27.711 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T02:45:27.721 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:27.726 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T02:45:27.760 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T02:45:27.878 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T02:45:27.895 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T02:45:27.936 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:45:28.105 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T02:45:28.121 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T02:45:28.188 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T02:45:28.231 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T02:45:28.250 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T02:45:28.289 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:45:28.441 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T02:45:28.456 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:28.500 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:28.621 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-webob. 2024-09-15T02:45:28.639 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T02:45:28.641 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T02:45:28.661 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T02:45:28.672 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:45:28.724 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:45:28.898 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-base. 2024-09-15T02:45:28.916 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:28.938 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T02:45:28.955 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T02:45:28.959 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:28.983 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T02:45:28.996 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:45:28.999 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T02:45:29.041 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:45:29.153 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T02:45:29.170 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T02:45:29.211 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T02:45:29.341 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T02:45:29.356 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T02:45:29.377 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T02:45:29.395 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T02:45:29.398 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:45:29.435 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T02:45:29.588 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T02:45:29.608 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T02:45:29.646 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T02:45:29.777 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T02:45:29.794 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T02:45:29.813 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T02:45:29.833 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T02:45:29.836 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T02:45:29.869 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T02:45:29.874 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:45:29.886 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T02:45:29.907 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T02:45:29.925 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:29.927 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:45:29.965 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:30.189 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T02:45:30.205 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T02:45:30.247 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:45:30.266 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T02:45:30.284 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T02:45:30.331 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T02:45:30.347 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T02:45:30.358 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T02:45:30.378 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T02:45:30.389 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T02:45:30.395 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T02:45:30.453 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:45:30.592 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T02:45:30.608 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T02:45:30.669 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-paste. 2024-09-15T02:45:30.675 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T02:45:30.685 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T02:45:30.741 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T02:45:30.752 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:45:30.759 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T02:45:30.789 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T02:45:30.800 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T02:45:30.807 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T02:45:30.847 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:45:31.012 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T02:45:31.028 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T02:45:31.053 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T02:45:31.069 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:31.072 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T02:45:31.078 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:45:31.088 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T02:45:31.111 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:31.139 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:45:31.209 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T02:45:31.226 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T02:45:31.309 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T02:45:31.423 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-mako. 2024-09-15T02:45:31.439 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T02:45:31.450 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T02:45:31.465 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T02:45:31.481 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:45:31.511 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T02:45:31.529 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T02:45:31.533 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:45:31.582 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:45:31.645 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T02:45:31.662 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T02:45:31.703 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:45:31.851 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T02:45:31.870 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T02:45:31.871 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T02:45:31.891 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T02:45:31.902 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T02:45:31.909 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:45:31.918 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T02:45:31.940 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T02:45:31.960 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:45:32.048 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T02:45:32.066 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T02:45:32.106 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T02:45:32.237 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T02:45:32.257 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T02:45:32.276 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T02:45:32.293 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T02:45:32.295 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:45:32.330 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T02:45:32.334 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:45:32.347 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T02:45:32.388 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T02:45:32.450 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T02:45:32.466 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T02:45:32.517 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:45:32.615 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webob. 2024-09-15T02:45:32.633 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T02:45:32.673 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:45:32.716 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T02:45:32.716 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T02:45:32.732 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T02:45:32.733 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T02:45:32.774 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:45:32.774 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:45:32.845 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-mako. 2024-09-15T02:45:32.860 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T02:45:32.903 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:45:33.001 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T02:45:33.021 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T02:45:33.059 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:45:33.128 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T02:45:33.147 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:33.186 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:33.248 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T02:45:33.263 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T02:45:33.331 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:45:33.396 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T02:45:33.411 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T02:45:33.487 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T02:45:33.565 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T02:45:33.583 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:33.623 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:33.659 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T02:45:33.675 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T02:45:33.712 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:45:33.822 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T02:45:33.841 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T02:45:33.878 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T02:45:33.894 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T02:45:33.923 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T02:45:33.929 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:45:34.137 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T02:45:34.158 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:34.195 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:34.231 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T02:45:34.248 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T02:45:34.289 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:45:34.634 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-mako. 2024-09-15T02:45:34.651 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T02:45:34.692 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:45:34.966 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-webob. 2024-09-15T02:45:34.982 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T02:45:35.000 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T02:45:35.019 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:35.024 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:45:35.028 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T02:45:35.049 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T02:45:35.086 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:45:35.145 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T02:45:35.160 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T02:45:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T02:45:35.383 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:35.385 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T02:45:35.402 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T02:45:35.406 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T02:45:35.425 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T02:45:35.464 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:45:35.468 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:45:35.514 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-paste. 2024-09-15T02:45:35.533 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T02:45:35.572 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:45:35.775 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webob. 2024-09-15T02:45:35.788 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T02:45:35.794 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T02:45:35.806 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T02:45:35.833 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:45:35.837 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T02:45:35.901 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T02:45:35.919 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T02:45:35.992 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:45:36.170 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T02:45:36.189 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T02:45:36.190 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T02:45:36.209 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T02:45:36.228 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:45:36.251 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:45:36.304 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T02:45:36.320 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T02:45:36.380 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:45:36.531 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T02:45:36.547 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T02:45:36.585 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T02:45:36.589 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T02:45:36.604 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T02:45:36.651 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T02:45:36.713 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T02:45:36.727 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T02:45:36.769 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T02:45:36.971 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-paste. 2024-09-15T02:45:36.975 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T02:45:36.990 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T02:45:36.992 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T02:45:37.029 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:45:37.036 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:45:37.105 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T02:45:37.123 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T02:45:37.164 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:45:37.349 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T02:45:37.362 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T02:45:37.366 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T02:45:37.381 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T02:45:37.407 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:45:37.420 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T02:45:37.543 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T02:45:37.559 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T02:45:37.601 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:45:37.748 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-paste. 2024-09-15T02:45:37.764 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T02:45:37.767 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T02:45:37.785 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T02:45:37.802 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:45:37.804 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph. 2024-09-15T02:45:37.820 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:37.838 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:45:37.862 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:37.980 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T02:45:37.996 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:38.038 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:38.123 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T02:45:38.145 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T02:45:38.145 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T02:45:38.157 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T02:45:38.181 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:45:38.199 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T02:45:38.207 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T02:45:38.224 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:38.265 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:38.466 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T02:45:38.485 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:38.509 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T02:45:38.524 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:38.527 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T02:45:38.544 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T02:45:38.559 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T02:45:38.575 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:45:38.602 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:45:38.653 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T02:45:38.671 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:38.728 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:38.878 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T02:45:38.896 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T02:45:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T02:45:39.005 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T02:45:39.022 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T02:45:39.063 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:45:39.087 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T02:45:39.103 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:39.187 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:39.290 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package cephadm. 2024-09-15T02:45:39.298 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T02:45:39.306 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:39.315 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T02:45:39.349 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:39.356 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:45:39.441 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T02:45:39.458 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:39.524 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:39.702 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T02:45:39.711 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T02:45:39.720 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T02:45:39.727 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T02:45:39.760 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:45:39.769 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:45:39.911 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T02:45:39.929 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:39.969 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:40.065 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T02:45:40.083 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:40.106 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T02:45:40.122 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:40.123 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:40.139 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T02:45:40.156 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:40.164 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:40.197 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:40.506 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T02:45:40.519 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T02:45:40.522 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:40.538 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T02:45:40.550 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T02:45:40.564 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:40.567 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:40.577 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T02:45:40.608 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:40.872 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T02:45:40.891 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T02:45:40.922 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:45:41.116 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T02:45:41.132 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:41.174 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:41.242 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T02:45:41.258 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T02:45:41.301 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:45:41.384 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T02:45:41.400 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:41.434 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:41.646 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-routes. 2024-09-15T02:45:41.663 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T02:45:41.704 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T02:45:41.986 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T02:45:42.003 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:42.043 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T02:45:42.043 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:42.062 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:42.101 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:42.310 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph. 2024-09-15T02:45:42.326 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:42.385 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T02:45:42.755 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:42.796 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:43.249 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T02:45:43.265 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:43.307 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:43.417 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T02:45:43.433 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T02:45:43.500 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:45:43.529 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph. 2024-09-15T02:45:43.546 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:43.595 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:43.787 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T02:45:43.806 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T02:45:43.860 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package cephadm. 2024-09-15T02:45:43.868 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:45:43.878 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:43.918 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:43.947 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T02:45:43.963 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:44.005 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:44.182 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T02:45:44.198 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T02:45:44.204 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph. 2024-09-15T02:45:44.220 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:44.254 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:44.271 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T02:45:44.273 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T02:45:44.288 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T02:45:44.330 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:45:44.409 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T02:45:44.429 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:44.492 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:44.506 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T02:45:44.523 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:44.564 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:44.691 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T02:45:44.707 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:44.749 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:44.984 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T02:45:45.004 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:45.051 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:45.134 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package cephadm. 2024-09-15T02:45:45.144 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T02:45:45.153 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:45.160 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T02:45:45.195 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:45.235 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T02:45:45.547 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T02:45:45.563 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T02:45:45.572 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T02:45:45.590 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T02:45:45.610 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T02:45:45.613 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package cephadm. 2024-09-15T02:45:45.628 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T02:45:45.630 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:45.630 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:45:45.630 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:45:45.668 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:45.671 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:45.916 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T02:45:45.934 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T02:45:45.966 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T02:45:45.974 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:45:45.985 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:45.990 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T02:45:46.008 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T02:45:46.024 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:46.049 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:45:46.248 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T02:45:46.265 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T02:45:46.298 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T02:45:46.310 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-routes. 2024-09-15T02:45:46.327 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T02:45:46.368 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T02:45:46.394 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T02:45:46.402 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T02:45:46.413 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T02:45:46.421 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:46.460 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:46.477 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T02:45:46.677 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T02:45:46.695 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T02:45:46.698 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T02:45:46.715 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:46.730 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T02:45:46.735 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:46.749 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T02:45:46.756 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:46.796 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T02:45:46.805 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:45:46.814 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T02:45:46.854 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T02:45:47.074 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T02:45:47.093 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T02:45:47.115 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T02:45:47.132 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:45:47.134 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T02:45:47.173 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:45:47.451 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T02:45:47.467 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T02:45:47.468 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-routes. 2024-09-15T02:45:47.486 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T02:45:47.497 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T02:45:47.500 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:45:47.513 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:47.527 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T02:45:47.555 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:47.804 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-routes. 2024-09-15T02:45:47.820 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T02:45:47.853 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T02:45:47.880 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T02:45:47.898 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:47.946 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:48.198 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T02:45:48.212 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T02:45:48.215 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:48.228 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T02:45:48.256 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:48.270 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:45:48.590 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T02:45:48.607 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T02:45:48.648 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:45:48.684 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T02:45:48.701 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T02:45:48.742 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T02:45:49.001 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T02:45:49.021 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T02:45:49.070 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T02:45:49.090 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T02:45:49.092 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T02:45:49.137 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:45:49.393 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T02:45:49.411 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T02:45:49.451 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:45:49.501 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T02:45:49.519 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T02:45:49.559 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:45:49.764 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T02:45:49.779 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T02:45:49.820 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:45:49.879 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T02:45:49.899 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T02:45:49.938 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T02:45:50.037 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T02:45:50.055 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T02:45:50.103 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:45:50.173 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T02:45:50.189 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T02:45:50.275 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T02:45:50.292 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T02:45:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T02:45:50.373 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T02:45:50.389 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T02:45:50.409 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T02:45:50.456 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:45:50.697 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T02:45:50.714 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T02:45:50.715 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T02:45:50.734 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T02:45:50.759 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T02:45:50.772 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:45:50.778 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T02:45:50.838 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T02:45:50.887 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:51.117 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T02:45:51.133 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T02:45:51.205 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:45:51.441 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T02:45:51.459 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T02:45:51.499 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T02:45:51.838 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T02:45:51.858 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T02:45:51.896 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:45:51.922 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T02:45:51.942 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T02:45:51.982 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T02:45:51.994 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:51.998 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T02:45:52.039 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:52.135 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T02:45:52.153 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T02:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:52.239 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T02:45:52.256 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T02:45:52.288 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:45:52.609 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T02:45:52.625 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T02:45:52.629 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T02:45:52.641 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T02:45:52.659 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T02:45:52.683 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:45:52.763 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T02:45:52.779 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T02:45:52.786 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T02:45:52.803 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:52.830 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T02:45:52.844 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:53.088 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T02:45:53.104 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T02:45:53.146 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:53.225 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T02:45:53.243 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T02:45:53.283 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:45:53.326 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T02:45:53.344 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:53.385 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:53.722 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T02:45:53.740 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T02:45:53.780 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:45:53.962 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T02:45:53.978 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:53.991 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T02:45:54.008 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:54.020 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:54.022 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T02:45:54.042 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T02:45:54.049 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:54.072 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T02:45:54.152 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T02:45:54.169 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:45:54.210 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:45:54.400 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T02:45:54.416 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T02:45:54.478 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:45:54.489 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package jq. 2024-09-15T02:45:54.505 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:45:54.547 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:45:54.832 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T02:45:54.833 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package socat. 2024-09-15T02:45:54.850 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T02:45:54.851 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T02:45:54.892 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T02:45:54.895 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:45:55.198 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T02:45:55.215 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T02:45:55.219 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T02:45:55.236 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T02:45:55.248 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T02:45:55.256 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T02:45:55.265 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T02:45:55.265 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T02:45:55.277 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T02:45:55.282 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T02:45:55.306 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:45:55.324 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T02:45:55.559 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T02:45:55.576 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T02:45:55.596 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T02:45:55.609 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:45:55.617 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T02:45:55.654 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:45:55.660 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T02:45:55.677 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T02:45:55.677 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-test. 2024-09-15T02:45:55.692 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:45:55.718 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T02:45:55.725 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:55.999 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T02:45:56.014 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T02:45:56.016 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T02:45:56.031 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T02:45:56.056 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T02:45:56.057 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:45:56.072 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:45:56.075 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T02:45:56.106 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:45:56.377 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T02:45:56.395 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T02:45:56.409 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T02:45:56.417 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T02:45:56.427 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T02:45:56.434 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T02:45:56.443 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T02:45:56.467 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:45:56.475 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T02:45:56.755 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T02:45:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T02:45:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T02:45:56.838 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T02:45:56.854 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T02:45:56.862 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T02:45:56.881 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T02:45:56.895 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:45:56.929 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T02:45:57.175 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T02:45:57.192 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T02:45:57.232 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:45:57.249 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T02:45:57.252 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T02:45:57.269 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T02:45:57.271 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T02:45:57.307 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:45:57.335 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:45:57.544 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T02:45:57.560 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T02:45:57.602 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:45:57.643 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T02:45:57.660 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T02:45:57.680 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T02:45:57.699 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T02:45:57.735 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:45:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:45:57.955 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T02:45:57.972 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T02:45:58.014 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:45:58.106 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T02:45:58.125 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T02:45:58.163 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:45:58.301 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T02:45:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T02:45:58.358 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:45:58.413 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T02:45:58.432 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:58.471 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:58.495 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T02:45:58.511 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T02:45:58.587 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:45:58.662 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T02:45:58.679 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T02:45:58.720 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:45:58.824 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T02:45:58.843 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T02:45:58.883 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:45:58.932 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T02:45:58.948 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T02:45:58.990 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:45:59.269 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T02:45:59.286 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:45:59.327 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:45:59.386 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T02:45:59.404 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:59.444 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:59.631 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T02:45:59.635 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package jq. 2024-09-15T02:45:59.650 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:45:59.654 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:45:59.689 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:45:59.693 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:45:59.781 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T02:45:59.801 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T02:45:59.841 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:45:59.987 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package socat. 2024-09-15T02:46:00.004 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T02:46:00.046 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T02:46:00.068 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T02:46:00.083 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T02:46:00.126 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:46:00.225 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T02:46:00.243 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:46:00.256 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T02:46:00.275 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:46:00.283 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:00.314 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:00.367 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T02:46:00.384 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T02:46:00.426 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:00.512 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T02:46:00.531 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:46:00.556 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package jq. 2024-09-15T02:46:00.575 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:46:00.596 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:00.614 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:00.670 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T02:46:00.690 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:00.728 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:00.874 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-test. 2024-09-15T02:46:00.884 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package socat. 2024-09-15T02:46:00.886 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package jq. 2024-09-15T02:46:00.891 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:00.902 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T02:46:00.905 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:46:00.932 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:00.942 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T02:46:00.944 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:01.231 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package socat. 2024-09-15T02:46:01.248 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T02:46:01.263 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T02:46:01.280 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T02:46:01.283 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T02:46:01.289 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T02:46:01.300 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:01.320 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:01.341 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:01.636 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T02:46:01.652 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T02:46:01.667 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-test. 2024-09-15T02:46:01.689 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:01.694 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:01.725 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:01.779 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T02:46:01.796 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T02:46:01.857 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:02.134 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-test. 2024-09-15T02:46:02.151 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:02.192 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:02.219 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T02:46:02.236 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:02.276 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:03.905 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T02:46:03.925 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T02:46:03.955 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:04.283 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T02:46:04.300 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T02:46:04.341 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:04.679 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-attr. 2024-09-15T02:46:04.695 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T02:46:04.736 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T02:46:05.107 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T02:46:05.122 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T02:46:05.163 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:05.370 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T02:46:05.390 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:46:05.436 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:05.474 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T02:46:05.495 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T02:46:05.532 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:05.783 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T02:46:05.802 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:05.841 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:05.903 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T02:46:05.919 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T02:46:05.961 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:06.395 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T02:46:06.412 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:06.456 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T02:46:06.461 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:06.473 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T02:46:06.507 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T02:46:06.514 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:06.523 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:46:06.565 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:06.867 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T02:46:06.881 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T02:46:06.887 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T02:46:06.890 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T02:46:06.899 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:06.907 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T02:46:06.925 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:06.939 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:06.947 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T02:46:06.964 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:46:06.991 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:07.030 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:07.270 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T02:46:07.287 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T02:46:07.328 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T02:46:07.329 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:07.347 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:07.351 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T02:46:07.370 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:07.378 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:07.409 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:07.461 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T02:46:07.479 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:07.518 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:07.690 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-py. 2024-09-15T02:46:07.708 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T02:46:07.748 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:07.921 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T02:46:07.941 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T02:46:07.997 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:08.033 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T02:46:08.050 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:08.088 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:08.126 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T02:46:08.143 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T02:46:08.184 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:08.328 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T02:46:08.345 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:08.386 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:08.479 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T02:46:08.496 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T02:46:08.534 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T02:46:08.537 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:08.552 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T02:46:08.609 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:08.865 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T02:46:08.884 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T02:46:08.934 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T02:46:08.980 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T02:46:08.999 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:46:09.063 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:09.177 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T02:46:09.196 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T02:46:09.234 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:09.288 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T02:46:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T02:46:09.346 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:09.604 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T02:46:09.615 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T02:46:09.623 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T02:46:09.633 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T02:46:09.662 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:09.707 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:09.973 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-attr. 2024-09-15T02:46:09.994 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T02:46:10.031 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T02:46:10.038 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T02:46:10.057 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T02:46:10.104 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:10.401 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T02:46:10.422 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T02:46:10.451 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:10.465 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T02:46:10.483 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T02:46:10.495 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package radosgw. 2024-09-15T02:46:10.511 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:10.532 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:10.553 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:10.754 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T02:46:10.774 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T02:46:10.779 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T02:46:10.798 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T02:46:10.829 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:10.834 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-attr. 2024-09-15T02:46:10.837 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:10.852 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T02:46:10.918 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T02:46:11.174 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T02:46:11.182 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T02:46:11.194 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T02:46:11.202 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T02:46:11.240 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:11.262 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T02:46:11.273 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:11.282 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T02:46:11.320 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:11.585 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-attr. 2024-09-15T02:46:11.602 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T02:46:11.623 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T02:46:11.640 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T02:46:11.659 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T02:46:11.697 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:11.777 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T02:46:11.794 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T02:46:11.835 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:12.004 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T02:46:12.017 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T02:46:12.021 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T02:46:12.038 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T02:46:12.062 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:12.075 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:12.189 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T02:46:12.205 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T02:46:12.247 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:12.348 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T02:46:12.368 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:12.390 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T02:46:12.406 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:12.408 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T02:46:12.448 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:12.545 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T02:46:12.563 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T02:46:12.591 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T02:46:12.603 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:12.609 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T02:46:12.691 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:12.748 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T02:46:12.843 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T02:46:12.860 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T02:46:12.901 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:12.932 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T02:46:12.948 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T02:46:12.990 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:13.047 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:46:13.061 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-py. 2024-09-15T02:46:13.078 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T02:46:13.119 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:13.173 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:46:13.324 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:13.343 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T02:46:13.360 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T02:46:13.396 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T02:46:13.401 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:13.413 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T02:46:13.454 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:13.464 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T02:46:13.480 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T02:46:13.522 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:13.525 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:13.737 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-py. 2024-09-15T02:46:13.754 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T02:46:13.795 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:13.800 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T02:46:13.816 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T02:46:13.858 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:13.862 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T02:46:13.879 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T02:46:13.920 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:14.122 INFO:teuthology.orchestra.run.smithi191.stdout:Adding system user cephadm....done 2024-09-15T02:46:14.148 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T02:46:14.168 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T02:46:14.206 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:14.253 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T02:46:14.264 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T02:46:14.272 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T02:46:14.281 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T02:46:14.283 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:14.310 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:14.323 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T02:46:14.435 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:46:14.543 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T02:46:14.560 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T02:46:14.601 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:14.655 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-py. 2024-09-15T02:46:14.672 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T02:46:14.685 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T02:46:14.706 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T02:46:14.713 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:14.772 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:14.828 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T02:46:14.912 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T02:46:14.931 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T02:46:14.970 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T02:46:15.046 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T02:46:15.064 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T02:46:15.109 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T02:46:15.126 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T02:46:15.129 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:15.145 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:15.167 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:15.280 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T02:46:15.307 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T02:46:15.325 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T02:46:15.365 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:15.406 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:15.548 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T02:46:15.565 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T02:46:15.606 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:15.651 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T02:46:15.671 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T02:46:15.709 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:15.733 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T02:46:15.958 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package radosgw. 2024-09-15T02:46:15.967 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T02:46:15.976 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:15.985 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T02:46:16.016 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:16.017 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T02:46:16.135 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:46:16.413 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T02:46:16.420 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T02:46:16.431 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T02:46:16.463 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package radosgw. 2024-09-15T02:46:16.470 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:16.483 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:16.521 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:16.705 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:46:16.802 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T02:46:16.820 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T02:46:16.865 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:17.014 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:17.323 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:17.624 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:17.653 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package radosgw. 2024-09-15T02:46:17.669 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:17.710 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:46:17.872 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T02:46:17.889 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:17.938 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:18.035 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:18.252 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T02:46:18.282 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T02:46:18.300 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:18.336 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:46:18.340 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:18.462 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:18.563 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:46:18.668 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T02:46:18.722 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:46:18.830 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T02:46:18.848 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:18.956 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:46:19.075 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:19.082 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:46:19.123 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:19.209 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:19.415 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:19.426 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:19.558 INFO:teuthology.orchestra.run.smithi152.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T02:46:19.578 INFO:teuthology.orchestra.run.smithi152.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:46:19.625 INFO:teuthology.orchestra.run.smithi152.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:19.646 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user cephadm....done 2024-09-15T02:46:19.783 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:46:19.850 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:19.940 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T02:46:19.976 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:46:19.988 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user cephadm....done 2024-09-15T02:46:20.101 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:46:20.191 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:20.238 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:46:20.317 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:46:20.361 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T02:46:20.364 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:46:20.435 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T02:46:20.515 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:46:20.679 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:20.720 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T02:46:20.742 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:20.822 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T02:46:20.929 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T02:46:20.948 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:21.038 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:46:21.164 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:21.206 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:21.275 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T02:46:21.290 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:21.331 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T02:46:21.458 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:21.583 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:46:21.784 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T02:46:21.903 INFO:teuthology.orchestra.run.smithi152.stdout:Adding system user cephadm....done 2024-09-15T02:46:22.075 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:22.186 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:46:22.218 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:46:22.394 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:46:22.487 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T02:46:22.526 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:46:22.561 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T02:46:22.706 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T02:46:22.747 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:46:22.854 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:46:22.902 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:46:22.980 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T02:46:22.996 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:46:23.031 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:23.140 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T02:46:23.279 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T02:46:23.314 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:23.334 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:23.405 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:23.467 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T02:46:23.618 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:23.632 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:23.723 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:23.744 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:46:23.849 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:23.869 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:46:23.925 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:24.037 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:24.050 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:46:24.201 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:46:24.237 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T02:46:24.305 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:46:24.360 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:24.369 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:24.406 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:24.494 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:24.529 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:46:24.695 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:46:24.790 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T02:46:24.805 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T02:46:24.821 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:24.854 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:25.083 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:25.107 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:25.164 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:25.239 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T02:46:25.289 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:46:25.368 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:25.408 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:46:25.408 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:25.534 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:46:25.582 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:25.711 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:46:25.734 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:46:25.852 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T02:46:25.891 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:25.995 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:46:26.018 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T02:46:26.169 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:46:26.251 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:46:26.311 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:46:26.321 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T02:46:26.337 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T02:46:26.437 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:46:26.535 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:46:26.597 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T02:46:26.722 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T02:46:26.730 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T02:46:26.890 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:46:26.922 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T02:46:27.024 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:27.048 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T02:46:27.056 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:46:27.184 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:27.213 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T02:46:27.357 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:46:27.385 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T02:46:27.468 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:46:27.589 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:46:27.729 INFO:teuthology.orchestra.run.smithi191.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T02:46:27.740 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:27.752 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:46:27.866 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:27.870 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:46:27.993 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:46:28.037 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:46:28.119 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:46:28.150 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:46:28.238 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:46:28.372 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T02:46:28.471 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:46:28.561 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:46:28.594 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T02:46:28.664 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T02:46:28.724 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:28.855 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:46:28.921 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:46:28.966 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:46:29.015 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:46:29.059 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:29.117 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T02:46:29.141 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:46:29.193 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:29.243 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T02:46:29.289 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T02:46:29.449 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:29.450 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:46:29.528 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:46:29.586 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:46:29.704 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:29.760 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:29.775 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:29.805 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:29.886 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T02:46:29.909 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:29.929 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:46:30.090 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:30.253 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:46:30.314 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:46:30.379 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:30.441 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:30.530 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:30.567 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:46:30.683 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T02:46:30.693 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:46:30.839 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:30.842 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:31.020 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:46:31.161 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:31.166 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:31.292 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:46:31.295 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T02:46:31.312 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T02:46:31.375 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:31.418 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:46:31.476 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:31.577 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T02:46:31.597 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:46:31.697 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:46:31.739 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:46:31.823 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:46:31.945 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T02:46:31.973 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T02:46:32.016 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T02:46:32.063 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T02:46:32.247 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:46:32.300 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T02:46:32.317 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:46:32.322 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:46:32.448 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:46:32.573 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:32.618 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:32.626 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:46:32.699 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T02:46:32.744 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:46:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T02:46:32.904 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:46:33.028 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T02:46:33.033 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:33.042 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:46:33.143 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:46:33.239 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:46:33.287 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:46:33.360 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:46:33.396 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:46:33.478 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:33.532 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T02:46:33.604 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:46:33.690 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:46:33.767 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T02:46:33.833 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:46:33.939 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:33.959 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:46:34.024 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:46:34.038 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:46:34.159 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:46:34.164 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:46:34.293 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:46:34.302 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T02:46:34.482 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:46:34.603 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:46:34.636 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:46:34.858 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:46:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:46:34.945 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:34.984 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:46:35.011 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:35.071 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T02:46:35.137 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:46:35.144 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:46:35.263 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:35.277 INFO:teuthology.orchestra.run.smithi152.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T02:46:35.414 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:35.479 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:46:35.532 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T02:46:35.557 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:46:35.676 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:46:35.772 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:35.833 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:46:35.914 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T02:46:35.959 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T02:46:36.043 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:46:36.302 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:36.412 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:46:36.450 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:36.538 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:46:36.552 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:36.669 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:36.669 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:36.678 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:36.689 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:46:36.804 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T02:46:37.023 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:46:37.121 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:37.189 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T02:46:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:37.354 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:37.357 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:46:37.473 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:46:37.493 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T02:46:37.513 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:37.639 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T02:46:37.667 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:37.758 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:37.911 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T02:46:37.994 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:46:38.008 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T02:46:38.262 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:38.309 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:46:38.380 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:38.388 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:46:38.531 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:38.602 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:38.640 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:38.690 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:38.728 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T02:46:38.766 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:38.808 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:46:38.934 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T02:46:38.965 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:39.063 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:39.092 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:39.142 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:39.180 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:46:39.251 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T02:46:39.263 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T02:46:39.532 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:39.579 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:39.620 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T02:46:39.683 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:46:39.738 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:39.864 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:46:39.921 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:46:40.001 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:40.205 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:46:40.215 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:40.316 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:46:40.365 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T02:46:40.442 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:40.568 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:40.694 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T02:46:40.716 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:40.775 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:40.834 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:46:40.987 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:46:41.110 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:41.117 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T02:46:41.229 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:41.246 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:46:41.346 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:46:41.392 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:41.396 INFO:teuthology.orchestra.run.smithi191.stdout:Adding group ceph....done 2024-09-15T02:46:41.398 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:41.501 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:41.519 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:41.645 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T02:46:41.688 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:46:41.710 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:41.710 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:41.763 INFO:teuthology.orchestra.run.smithi191.stdout:Adding system user ceph....done 2024-09-15T02:46:41.939 INFO:teuthology.orchestra.run.smithi191.stdout:Setting system user ceph properties....done 2024-09-15T02:46:41.957 INFO:teuthology.orchestra.run.smithi191.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T02:46:41.963 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:46:42.097 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T02:46:42.098 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T02:46:42.200 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:42.423 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T02:46:42.432 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:42.436 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:46:42.558 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:42.705 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:42.784 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:42.784 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:42.795 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:42.939 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:42.946 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:46:43.006 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:46:43.062 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:43.072 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T02:46:43.209 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:43.223 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:46:43.349 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T02:46:43.350 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:43.360 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:43.360 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:43.421 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:46:43.476 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:43.628 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:43.659 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:46:43.723 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:43.754 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:43.818 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T02:46:43.881 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T02:46:44.014 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.066 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:46:44.136 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.206 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T02:46:44.262 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.399 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T02:46:44.427 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.494 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.515 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:44.515 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T02:46:44.617 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.628 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.754 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.768 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:46:44.874 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.914 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:44.955 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:45.040 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T02:46:45.108 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:45.108 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:45.159 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:46:45.341 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T02:46:45.477 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T02:46:45.635 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:45.702 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:45.721 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:45.769 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:45.846 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:46:45.895 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:46:45.933 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:45.933 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:46.182 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:Adding group ceph....done 2024-09-15T02:46:46.308 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:46.411 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:46.476 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:46.602 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:46.651 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user ceph....done 2024-09-15T02:46:46.753 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T02:46:46.824 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:46.827 INFO:teuthology.orchestra.run.smithi006.stdout:Setting system user ceph properties....done 2024-09-15T02:46:46.854 INFO:teuthology.orchestra.run.smithi006.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T02:46:46.900 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:46.900 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:47.010 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T02:46:47.071 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T02:46:47.329 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T02:46:47.346 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.365 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.427 INFO:teuthology.orchestra.run.smithi118.stdout:Adding group ceph....done 2024-09-15T02:46:47.472 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.482 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.615 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.642 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:46:47.793 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.812 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user ceph....done 2024-09-15T02:46:47.823 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:47.823 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:47.924 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:47.988 INFO:teuthology.orchestra.run.smithi118.stdout:Setting system user ceph properties....done 2024-09-15T02:46:48.012 INFO:teuthology.orchestra.run.smithi118.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T02:46:48.155 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T02:46:48.202 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:48.202 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:48.267 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:48.452 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:48.484 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T02:46:48.546 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:48.578 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:48.746 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:48.873 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T02:46:48.876 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:48.918 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:49.024 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T02:46:49.044 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:49.091 INFO:teuthology.orchestra.run.smithi152.stdout:Adding group ceph....done 2024-09-15T02:46:49.314 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T02:46:49.334 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T02:46:49.364 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:49.364 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:49.510 INFO:teuthology.orchestra.run.smithi152.stdout:Adding system user ceph....done 2024-09-15T02:46:49.678 INFO:teuthology.orchestra.run.smithi152.stdout:Setting system user ceph properties....done 2024-09-15T02:46:49.702 INFO:teuthology.orchestra.run.smithi152.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T02:46:49.848 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T02:46:49.879 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:50.004 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:50.113 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:50.113 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:50.154 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T02:46:50.414 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T02:46:50.547 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:50.574 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:50.702 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:50.785 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:50.785 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:50.881 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:50.993 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:50.993 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:46:51.120 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:51.121 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:51.232 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:51.575 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:51.611 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:51.727 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:51.727 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:51.805 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:51.806 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:51.974 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T02:46:52.177 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:52.226 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:52.278 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T02:46:52.301 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:52.475 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:52.478 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:52.816 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:52.816 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:52.927 INFO:teuthology.orchestra.run.smithi191.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T02:46:52.932 DEBUG:teuthology.parallel:result is None 2024-09-15T02:46:53.119 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:53.119 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T02:46:53.179 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.242 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.322 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.388 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.439 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.540 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.555 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.582 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:53.709 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T02:46:53.757 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:53.757 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:53.782 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:53.782 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T02:46:53.860 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T02:46:54.137 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T02:46:54.199 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:54.208 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:54.342 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:54.543 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:54.686 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:54.716 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:54.716 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T02:46:54.838 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T02:46:54.990 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T02:46:55.145 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:55.258 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T02:46:55.269 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:55.446 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:55.670 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:55.670 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T02:46:56.080 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:56.206 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:56.366 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:56.509 INFO:teuthology.orchestra.run.smithi152.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:46:56.669 INFO:teuthology.orchestra.run.smithi152.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T02:46:56.879 INFO:teuthology.orchestra.run.smithi152.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T02:46:57.024 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T02:46:57.173 INFO:teuthology.orchestra.run.smithi152.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T02:46:57.619 INFO:teuthology.orchestra.run.smithi006.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T02:46:57.623 DEBUG:teuthology.parallel:result is None 2024-09-15T02:46:58.053 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T02:46:58.679 INFO:teuthology.orchestra.run.smithi118.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T02:46:58.683 DEBUG:teuthology.parallel:result is None 2024-09-15T02:47:00.076 INFO:teuthology.orchestra.run.smithi152.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T02:47:00.701 INFO:teuthology.orchestra.run.smithi152.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T02:47:00.705 DEBUG:teuthology.parallel:result is None 2024-09-15T02:47:00.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:47:00.852 DEBUG:teuthology.orchestra.run.smithi006:> dpkg-query -W -f '${Version}' ceph 2024-09-15T02:47:00.875 INFO:teuthology.orchestra.run.smithi006.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:00.875 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:00.876 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T02:47:00.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:47:01.042 DEBUG:teuthology.orchestra.run.smithi118:> dpkg-query -W -f '${Version}' ceph 2024-09-15T02:47:01.065 INFO:teuthology.orchestra.run.smithi118.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:01.065 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:01.066 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T02:47:01.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:47:01.220 DEBUG:teuthology.orchestra.run.smithi152:> dpkg-query -W -f '${Version}' ceph 2024-09-15T02:47:01.245 INFO:teuthology.orchestra.run.smithi152.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:01.245 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:01.245 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T02:47:01.247 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:47:01.396 DEBUG:teuthology.orchestra.run.smithi191:> dpkg-query -W -f '${Version}' ceph 2024-09-15T02:47:01.419 INFO:teuthology.orchestra.run.smithi191.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:01.419 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:47:01.419 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T02:47:01.421 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T02:47:01.421 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:01.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:47:01.438 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:01.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:47:01.455 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:01.455 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:47:01.472 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:01.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:47:01.488 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T02:47:01.488 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:01.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:47:01.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:47:01.563 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:01.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:47:01.579 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:47:01.634 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:01.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:47:01.651 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:47:01.666 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:01.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:47:01.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:47:01.741 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T02:47:02.410 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:02.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:47:02.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:47:02.483 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:02.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:47:02.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:47:02.509 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:02.510 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:47:02.527 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:47:02.541 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:02.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:47:02.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:47:02.567 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T02:47:02.568 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:02.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:47:02.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:47:02.594 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:02.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:47:02.611 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:47:02.623 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:02.623 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:47:02.638 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:47:02.649 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:02.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:47:02.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:47:02.679 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T02:47:02.912 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T02:47:02.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-09-15T02:47:02.915 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-09-15T02:47:02.917 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-09-15T02:47:02.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-09-15T02:47:02.939 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T02:47:02.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T02:47:02.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T02:47:02.973 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T02:47:02.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T02:47:02.997 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T02:47:02.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T02:47:02.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T02:47:03.002 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T02:47:03.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T02:47:03.034 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T02:47:03.034 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T02:47:03.034 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905171', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T02:47:03.035 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T02:47:03.058 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T02:47:03.060 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T02:47:03.062 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T02:47:03.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T02:47:03.071 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T02:47:03.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T02:47:03.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T02:47:03.130 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:03.131 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T02:47:03.137 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T02:47:03.138 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-15 02:43:18.571491653 +0000 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-15 02:43:18.351496724 +0000 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-15 02:43:18.351496724 +0000 2024-09-15T02:47:03.145 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-15T02:47:03.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T02:47:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-15T02:47:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-15T02:47:03.159 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000433217 s, 1.2 MB/s 2024-09-15T02:47:03.159 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T02:47:03.212 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-09-15T02:47:03.218 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T02:47:03.219 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.219 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 366 Links: 1 2024-09-15T02:47:03.219 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.219 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-15 02:43:18.995481880 +0000 2024-09-15T02:47:03.220 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-15 02:43:18.795486491 +0000 2024-09-15T02:47:03.220 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-15 02:43:18.795486491 +0000 2024-09-15T02:47:03.220 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-15T02:47:03.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T02:47:03.232 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-15T02:47:03.232 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-15T02:47:03.232 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000422534 s, 1.2 MB/s 2024-09-15T02:47:03.233 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T02:47:03.284 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-15 02:43:19.399472569 +0000 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-15 02:43:19.187477455 +0000 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-15 02:43:19.187477455 +0000 2024-09-15T02:47:03.291 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-15T02:47:03.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T02:47:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-15T02:47:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-15T02:47:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000410947 s, 1.2 MB/s 2024-09-15T02:47:03.304 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T02:47:03.357 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 378 Links: 1 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-15 02:43:19.859461966 +0000 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-15 02:43:19.631467220 +0000 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-15 02:43:19.631467220 +0000 2024-09-15T02:47:03.364 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-15T02:47:03.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T02:47:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-15T02:47:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-15T02:47:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000365741 s, 1.4 MB/s 2024-09-15T02:47:03.378 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T02:47:03.386 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:03.387 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:03.387 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T02:47:03.394 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T02:47:03.394 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-09-15T02:47:03.401 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T02:47:03.401 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.401 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 358 Links: 1 2024-09-15T02:47:03.401 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.401 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-15 02:43:18.348780860 +0000 2024-09-15T02:47:03.401 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-15 02:43:18.344780951 +0000 2024-09-15T02:47:03.402 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-15 02:43:18.344780951 +0000 2024-09-15T02:47:03.402 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-15T02:47:03.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T02:47:03.458 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-15T02:47:03.458 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-15T02:47:03.458 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.00030371 s, 1.7 MB/s 2024-09-15T02:47:03.460 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T02:47:03.468 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-09-15T02:47:03.475 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T02:47:03.475 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.475 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 364 Links: 1 2024-09-15T02:47:03.475 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.475 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-15 02:43:18.760771358 +0000 2024-09-15T02:47:03.476 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-15 02:43:18.756771450 +0000 2024-09-15T02:47:03.476 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-15 02:43:18.756771450 +0000 2024-09-15T02:47:03.476 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-15T02:47:03.476 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T02:47:03.532 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-15T02:47:03.532 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-15T02:47:03.532 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000293947 s, 1.7 MB/s 2024-09-15T02:47:03.532 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T02:47:03.540 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-09-15T02:47:03.545 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T02:47:03.545 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.545 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-15T02:47:03.546 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.546 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-15 02:43:19.416756229 +0000 2024-09-15T02:47:03.546 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-15 02:43:19.200761212 +0000 2024-09-15T02:47:03.546 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-15 02:43:19.200761212 +0000 2024-09-15T02:47:03.546 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-15T02:47:03.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T02:47:03.602 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-15T02:47:03.602 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-15T02:47:03.602 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000288971 s, 1.8 MB/s 2024-09-15T02:47:03.603 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T02:47:03.610 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-09-15T02:47:03.616 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T02:47:03.616 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.616 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-15T02:47:03.616 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.616 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-15 02:43:19.852746176 +0000 2024-09-15T02:47:03.617 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-15 02:43:19.648750880 +0000 2024-09-15T02:47:03.617 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-15 02:43:19.648750880 +0000 2024-09-15T02:47:03.617 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-15T02:47:03.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T02:47:03.675 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-15T02:47:03.675 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-15T02:47:03.675 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000285964 s, 1.8 MB/s 2024-09-15T02:47:03.675 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T02:47:03.683 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:03.683 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:03.683 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T02:47:03.690 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T02:47:03.690 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-09-15T02:47:03.742 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T02:47:03.742 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.742 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-09-15T02:47:03.742 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.742 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-15 02:43:23.015088903 +0000 2024-09-15T02:47:03.742 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-15 02:43:22.831093138 +0000 2024-09-15T02:47:03.743 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-15 02:43:22.831093138 +0000 2024-09-15T02:47:03.743 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-09-15T02:47:03.743 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T02:47:03.754 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-15T02:47:03.755 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-15T02:47:03.755 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000419921 s, 1.2 MB/s 2024-09-15T02:47:03.756 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T02:47:03.804 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 364 Links: 1 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-15 02:43:23.483078131 +0000 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-15 02:43:23.263083195 +0000 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-15 02:43:23.263083195 +0000 2024-09-15T02:47:03.812 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-09-15T02:47:03.812 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T02:47:03.822 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-15T02:47:03.823 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-15T02:47:03.823 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000402585 s, 1.3 MB/s 2024-09-15T02:47:03.824 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T02:47:03.876 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-15 02:43:23.931067819 +0000 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-15 02:43:23.699073160 +0000 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-15 02:43:23.699073160 +0000 2024-09-15T02:47:03.883 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-09-15T02:47:03.883 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T02:47:03.895 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-15T02:47:03.895 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-15T02:47:03.895 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000412291 s, 1.2 MB/s 2024-09-15T02:47:03.896 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T02:47:03.955 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-15 02:43:24.359057967 +0000 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-15 02:43:24.143062939 +0000 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-15 02:43:24.143062939 +0000 2024-09-15T02:47:03.962 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-09-15T02:47:03.963 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T02:47:04.018 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-15T02:47:04.019 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-15T02:47:04.019 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000378722 s, 1.4 MB/s 2024-09-15T02:47:04.019 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T02:47:04.028 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:04.028 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:04.028 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T02:47:04.035 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T02:47:04.035 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-09-15T02:47:04.042 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T02:47:04.042 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:04.042 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 358 Links: 1 2024-09-15T02:47:04.043 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:04.043 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-15 02:43:23.823179504 +0000 2024-09-15T02:47:04.043 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-15 02:43:23.819179595 +0000 2024-09-15T02:47:04.043 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-15 02:43:23.819179595 +0000 2024-09-15T02:47:04.043 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2024-09-15T02:47:04.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T02:47:04.054 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-15T02:47:04.054 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-15T02:47:04.054 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000422821 s, 1.2 MB/s 2024-09-15T02:47:04.055 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T02:47:04.063 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 364 Links: 1 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-15 02:43:24.447165147 +0000 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-15 02:43:24.247169748 +0000 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-15 02:43:24.247169748 +0000 2024-09-15T02:47:04.112 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2024-09-15T02:47:04.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T02:47:04.123 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-15T02:47:04.123 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-15T02:47:04.123 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000397639 s, 1.3 MB/s 2024-09-15T02:47:04.124 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T02:47:04.131 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-09-15T02:47:04.136 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T02:47:04.136 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:04.136 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-15T02:47:04.136 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:04.137 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-15 02:43:24.899154748 +0000 2024-09-15T02:47:04.137 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-15 02:43:24.671159994 +0000 2024-09-15T02:47:04.137 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-15 02:43:24.671159994 +0000 2024-09-15T02:47:04.137 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2024-09-15T02:47:04.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T02:47:04.148 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-15T02:47:04.148 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-15T02:47:04.149 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000415618 s, 1.2 MB/s 2024-09-15T02:47:04.150 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T02:47:04.201 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 380 Links: 1 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-15 02:43:25.527140299 +0000 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-15 02:43:25.287145822 +0000 2024-09-15T02:47:04.209 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-15 02:43:25.287145822 +0000 2024-09-15T02:47:04.210 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2024-09-15T02:47:04.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T02:47:04.222 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-15T02:47:04.223 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-15T02:47:04.223 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000361386 s, 1.4 MB/s 2024-09-15T02:47:04.223 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T02:47:04.273 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:04.273 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-15T02:47:04.273 INFO:tasks.ceph:Generating config... 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T02:47:04.275 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T02:47:04.276 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T02:47:04.277 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-15T02:47:04.277 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-15T02:47:04.277 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T02:47:04.277 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-15T02:47:04.277 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T02:47:04.277 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T02:47:04.278 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T02:47:04.278 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T02:47:04.278 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T02:47:04.278 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T02:47:04.278 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T02:47:04.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T02:47:04.307 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T02:47:04.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T02:47:04.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:47:04.406 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.118'), ('mon.c', '172.21.15.152')] 2024-09-15T02:47:04.407 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.118,172.21.15.152', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-15T02:47:04.408 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:04.408 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T02:47:04.415 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.118 --add c 172.21.15.152 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:47:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = octopus 2024-09-15T02:47:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-09-15T02:47:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:47:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-09-15T02:47:04.451524+0000 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-15T02:47:04.451524+0000 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 15 (octopus) 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 3 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.c 2024-09-15T02:47:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T02:47:04.455 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T02:47:04.460 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 24e12693-a4e4-4f9a-bb1d-999f41be75a2... 2024-09-15T02:47:04.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T02:47:04.507 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T02:47:04.509 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T02:47:04.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T02:47:04.540 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-15T02:47:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-15T02:47:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.118,172.21.15.152" 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon election default strategy = 3 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay max = 1 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay probability = 0.005 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay type = osd 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject internal delays = 0.002 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 2500 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:47:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 120 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-15T02:47:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-15T02:47:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = zlib 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore zero block detection = True 2024-09-15T02:47:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug reject backfill probability = 0.1 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify cached snaps = True 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify missing on start = True 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd max backfills = 3 2024-09-15T02:47:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd max markdown count = 1000 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock override recovery settings = True 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue = debug_random 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue cut off = debug_random 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub min interval = 60 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd shutdown pgref assert = True 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-15T02:47:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub interval = 300 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-09-15T02:47:04.552 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.554 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-09-15T02:47:04.555 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-09-15T02:47:04.556 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.6,172.21.15.118,172.21.15.152" 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: mon election default strategy = 3 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay max = 1 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay probability = 0.005 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay type = osd 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject internal delays = 0.002 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 2500 2024-09-15T02:47:04.557 INFO:teuthology.orchestra.run.smithi191.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 120 2024-09-15T02:47:04.558 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-09-15T02:47:04.559 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-09-15T02:47:04.560 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression algorithm = zlib 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore zero block detection = True 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug reject backfill probability = 0.1 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify cached snaps = True 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify missing on start = True 2024-09-15T02:47:04.561 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd max backfills = 3 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock override recovery settings = True 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue = debug_random 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue cut off = debug_random 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub min interval = 60 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: osd shutdown pgref assert = True 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-15T02:47:04.562 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-09-15T02:47:04.563 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-09-15T02:47:04.564 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub interval = 300 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2024-09-15T02:47:04.565 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2024-09-15T02:47:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-09-15T02:47:04.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-09-15T02:47:04.568 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-09-15T02:47:04.569 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = "172.21.15.6,172.21.15.118,172.21.15.152" 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: mon election default strategy = 3 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject delay max = 1 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject delay probability = 0.005 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject delay type = osd 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject internal delays = 0.002 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 2500 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-09-15T02:47:04.570 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 120 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.571 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 20 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression algorithm = zlib 2024-09-15T02:47:04.572 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression mode = aggressive 2024-09-15T02:47:04.573 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-09-15T02:47:04.573 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore zero block detection = True 2024-09-15T02:47:04.573 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 20 2024-09-15T02:47:04.573 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 20 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 10 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug reject backfill probability = 0.1 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify cached snaps = True 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify missing on start = True 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd max backfills = 3 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd max markdown count = 1000 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock override recovery settings = True 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-09-15T02:47:04.574 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue = debug_random 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue cut off = debug_random 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub min interval = 60 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: osd shutdown pgref assert = True 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.575 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T02:47:04.576 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: mon scrub interval = 300 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-09-15T02:47:04.577 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.b] 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.c] 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-09-15T02:47:04.578 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-09-15T02:47:04.579 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.580 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "172.21.15.6,172.21.15.118,172.21.15.152" 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: mon client directed command retry = 5 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: mon election default strategy = 3 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject delay max = 1 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject delay probability = 0.005 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject delay type = osd 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject internal delays = 0.002 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject socket failures = 2500 2024-09-15T02:47:04.581 INFO:teuthology.orchestra.run.smithi118.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 120 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-09-15T02:47:04.582 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-09-15T02:47:04.583 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression algorithm = zlib 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression mode = aggressive 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore zero block detection = True 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 20 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 20 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 10 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2024-09-15T02:47:04.584 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug reject backfill probability = 0.1 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify cached snaps = True 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify missing on start = True 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd max backfills = 3 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd max markdown count = 1000 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock override recovery settings = True 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue = debug_random 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue cut off = debug_random 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub min interval = 60 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd shutdown pgref assert = True 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-09-15T02:47:04.585 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-09-15T02:47:04.586 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-09-15T02:47:04.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub interval = 300 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2024-09-15T02:47:04.588 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2024-09-15T02:47:04.589 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T02:47:04.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T02:47:04.640 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T02:47:04.640 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:04.640 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T02:47:04.647 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:04.647 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T02:47:04.652 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:47:04.652 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:04.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T02:47:04.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:47:04.678 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:47:04.678 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:04.683 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:47:04.683 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:04.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T02:47:04.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:47:04.708 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:47:04.708 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:04.713 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:47:04.713 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:04.713 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T02:47:04.713 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:47:04.740 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:47:04.741 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:04.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:47:04.745 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:04.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T02:47:04.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T02:47:04.772 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:47:04.772 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:04.776 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T02:47:04.776 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T02:47:04.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-15T02:47:04.823 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-15T02:47:04.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T02:47:04.871 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T02:47:04.873 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T02:47:04.873 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T02:47:04.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T02:47:04.907 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T02:47:04.920 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T02:47:04.920 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-15T02:47:04.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T02:47:04.938 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:04.938 INFO:tasks.ceph:role: osd.0 2024-09-15T02:47:04.939 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:47:04.939 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T02:47:05.458 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 2024-09-15T02:47:05.458 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:05.459 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:05.459 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:05.459 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:05.459 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:05.460 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-15T02:47:05.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T02:47:05.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T02:47:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:05.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T02:47:05.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:05.561 INFO:tasks.ceph:role: osd.4 2024-09-15T02:47:05.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:47:05.561 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T02:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 2024-09-15T02:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:06.135 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-15T02:47:06.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T02:47:06.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T02:47:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:06.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-15T02:47:06.198 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:06.198 INFO:tasks.ceph:role: osd.8 2024-09-15T02:47:06.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:47:06.198 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T02:47:06.669 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:06.669 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:06.669 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:06.669 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 2024-09-15T02:47:06.669 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:06.670 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:06.670 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:06.670 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:06.670 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:06.670 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:06.672 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-15T02:47:06.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-15T02:47:06.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-15T02:47:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:06.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-15T02:47:06.748 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:06.748 INFO:tasks.ceph:role: osd.12 2024-09-15T02:47:06.748 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:47:06.748 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:07.260 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:07.262 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-15T02:47:07.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-15T02:47:07.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-15T02:47:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:07.326 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:07.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.396+0000 7fcdd25e9540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T02:47:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.396+0000 7fcdd25e9540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T02:47:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.396+0000 7fcdd25e9540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T02:47:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.396+0000 7fcdd25e9540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T02:47:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.396+0000 7fcdd25e9540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T02:47:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.396+0000 7fcdd25e9540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T02:47:07.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:07.960+0000 7fcdd25e9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:08.740+0000 7fcdd25e9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:09.520+0000 7fcdd25e9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:10.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T02:47:10.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:10.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.416+0000 7f09ad30e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T02:47:10.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.416+0000 7f09ad30e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T02:47:10.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.420+0000 7f09ad30e540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T02:47:10.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.420+0000 7f09ad30e540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T02:47:10.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.420+0000 7f09ad30e540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T02:47:10.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.420+0000 7f09ad30e540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T02:47:10.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:10.972+0000 7f09ad30e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:11.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:11.756+0000 7f09ad30e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:12.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:12.536+0000 7f09ad30e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:13.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T02:47:13.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:13.484+0000 7f9970863540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-15T02:47:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:13.484+0000 7f9970863540 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-15T02:47:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:13.484+0000 7f9970863540 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-09-15T02:47:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:13.484+0000 7f9970863540 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-09-15T02:47:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:13.484+0000 7f9970863540 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-09-15T02:47:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:13.488+0000 7f9970863540 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-15T02:47:14.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:14.048+0000 7f9970863540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:14.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:14.828+0000 7f9970863540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:15.620+0000 7f9970863540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:16.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-15T02:47:16.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:16.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:16.532+0000 7f0ec2b2c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-15T02:47:16.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:16.532+0000 7f0ec2b2c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-15T02:47:16.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:16.532+0000 7f0ec2b2c540 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-09-15T02:47:16.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:16.532+0000 7f0ec2b2c540 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-09-15T02:47:16.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:16.532+0000 7f0ec2b2c540 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-09-15T02:47:16.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:16.532+0000 7f0ec2b2c540 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-15T02:47:17.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:17.096+0000 7f0ec2b2c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:17.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:17.876+0000 7f0ec2b2c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:18.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:47:18.660+0000 7f0ec2b2c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:19.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-15T02:47:19.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T02:47:19.585 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:19.585 INFO:tasks.ceph:role: osd.1 2024-09-15T02:47:19.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:47:19.585 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:19.630 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:19.631 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:19.631 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:19.631 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:19.632 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-15T02:47:19.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-15T02:47:19.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T02:47:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:19.705 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:19.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T02:47:19.720 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:19.720 INFO:tasks.ceph:role: osd.5 2024-09-15T02:47:19.720 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:47:19.720 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T02:47:19.761 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:19.762 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:19.763 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-15T02:47:19.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-15T02:47:19.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T02:47:19.835 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:19.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-15T02:47:19.852 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:19.853 INFO:tasks.ceph:role: osd.9 2024-09-15T02:47:19.853 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:47:19.853 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T02:47:19.896 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2024-09-15T02:47:19.897 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:19.897 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:19.897 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:19.897 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:19.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-15T02:47:19.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-15T02:47:19.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-15T02:47:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:19.971 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:19.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-15T02:47:19.983 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:19.983 INFO:tasks.ceph:role: osd.13 2024-09-15T02:47:19.983 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:47:19.983 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:20.023 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:20.024 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:20.024 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:20.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-15T02:47:20.025 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-15T02:47:20.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-15T02:47:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:20.098 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:20.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:20.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.171+0000 7f234b14e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T02:47:20.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.171+0000 7f234b14e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T02:47:20.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.171+0000 7f234b14e540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T02:47:20.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.171+0000 7f234b14e540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T02:47:20.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.171+0000 7f234b14e540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T02:47:20.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.175+0000 7f234b14e540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T02:47:20.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:20.735+0000 7f234b14e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:21.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:21.523+0000 7f234b14e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:22.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:22.307+0000 7f234b14e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:23.160 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T02:47:23.177 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:23.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.211+0000 7fc2ce305540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T02:47:23.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.211+0000 7fc2ce305540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T02:47:23.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.211+0000 7fc2ce305540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T02:47:23.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.211+0000 7fc2ce305540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T02:47:23.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.211+0000 7fc2ce305540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T02:47:23.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.211+0000 7fc2ce305540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T02:47:23.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:23.767+0000 7fc2ce305540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:24.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:24.567+0000 7fc2ce305540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:25.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:25.347+0000 7fc2ce305540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:26.195 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T02:47:26.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:26.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.287+0000 7f26ee591540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-15T02:47:26.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.287+0000 7f26ee591540 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-15T02:47:26.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.287+0000 7f26ee591540 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-09-15T02:47:26.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.287+0000 7f26ee591540 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-09-15T02:47:26.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.287+0000 7f26ee591540 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-09-15T02:47:26.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.291+0000 7f26ee591540 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-15T02:47:26.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:26.843+0000 7f26ee591540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:27.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:27.627+0000 7f26ee591540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:28.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:28.407+0000 7f26ee591540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:29.271 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-15T02:47:29.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:29.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.323+0000 7fb4c7d33540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-15T02:47:29.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.323+0000 7fb4c7d33540 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-15T02:47:29.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.323+0000 7fb4c7d33540 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-09-15T02:47:29.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.323+0000 7fb4c7d33540 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-09-15T02:47:29.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.323+0000 7fb4c7d33540 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-09-15T02:47:29.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.327+0000 7fb4c7d33540 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-15T02:47:29.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:29.875+0000 7fb4c7d33540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:30.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:30.651+0000 7fb4c7d33540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:31.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-15T02:47:31.435+0000 7fb4c7d33540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:32.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-15T02:47:32.356 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T02:47:32.372 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:32.372 INFO:tasks.ceph:role: osd.2 2024-09-15T02:47:32.372 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:47:32.373 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T02:47:32.940 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:32.965 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:32.966 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-15T02:47:32.966 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-15T02:47:33.060 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T02:47:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:33.073 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:33.073 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-15T02:47:33.133 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:33.133 INFO:tasks.ceph:role: osd.6 2024-09-15T02:47:33.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:47:33.133 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T02:47:33.632 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:33.632 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:33.632 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:33.632 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 2024-09-15T02:47:33.632 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:33.633 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:33.633 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:33.633 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:33.633 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:33.633 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:33.634 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-15T02:47:33.635 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-15T02:47:33.667 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-15T02:47:33.720 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:33.722 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-15T02:47:33.733 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:33.733 INFO:tasks.ceph:role: osd.10 2024-09-15T02:47:33.733 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:47:33.734 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:34.215 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:34.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-15T02:47:34.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-15T02:47:34.272 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-15T02:47:34.285 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:34.286 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-15T02:47:34.343 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:34.343 INFO:tasks.ceph:role: osd.14 2024-09-15T02:47:34.343 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:47:34.343 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T02:47:34.863 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:34.863 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:34.863 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:34.866 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-15T02:47:34.866 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-15T02:47:34.925 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-15T02:47:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:34.938 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:34.938 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:35.045 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.039+0000 7fe2064b3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T02:47:35.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.039+0000 7fe2064b3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T02:47:35.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.039+0000 7fe2064b3540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T02:47:35.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.039+0000 7fe2064b3540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T02:47:35.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.039+0000 7fe2064b3540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T02:47:35.055 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.047+0000 7fe2064b3540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T02:47:35.627 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:35.619+0000 7fe2064b3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:36.409 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:36.403+0000 7fe2064b3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:37.194 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:37.187+0000 7fe2064b3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:38.037 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T02:47:38.053 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:38.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.079+0000 7eff5ff1e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-15T02:47:38.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.083+0000 7eff5ff1e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-15T02:47:38.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.083+0000 7eff5ff1e540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T02:47:38.089 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.083+0000 7eff5ff1e540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T02:47:38.090 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.083+0000 7eff5ff1e540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T02:47:38.090 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.083+0000 7eff5ff1e540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-15T02:47:38.641 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:38.635+0000 7eff5ff1e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:39.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:39.427+0000 7eff5ff1e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:40.215 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:40.206+0000 7eff5ff1e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:41.068 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-15T02:47:41.128 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:41.165 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.158+0000 7f7215b0b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-15T02:47:41.166 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.158+0000 7f7215b0b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-15T02:47:41.166 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.158+0000 7f7215b0b540 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-09-15T02:47:41.166 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.158+0000 7f7215b0b540 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-09-15T02:47:41.166 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.158+0000 7f7215b0b540 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-09-15T02:47:41.166 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.158+0000 7f7215b0b540 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-15T02:47:41.718 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:41.710+0000 7f7215b0b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:42.500 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:42.490+0000 7f7215b0b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:43.290 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:43.282+0000 7f7215b0b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:44.128 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-15T02:47:44.144 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:44.232 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.226+0000 7f07a82d6540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-15T02:47:44.232 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.226+0000 7f07a82d6540 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-15T02:47:44.233 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.226+0000 7f07a82d6540 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-09-15T02:47:44.233 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.226+0000 7f07a82d6540 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-09-15T02:47:44.233 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.226+0000 7f07a82d6540 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-09-15T02:47:44.233 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.230+0000 7f07a82d6540 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-15T02:47:44.791 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:44.786+0000 7f07a82d6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:45.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:45.578+0000 7f07a82d6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:46.366 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:47:46.362+0000 7f07a82d6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:47.212 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-15T02:47:47.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T02:47:47.241 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:47.241 INFO:tasks.ceph:role: osd.3 2024-09-15T02:47:47.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:47:47.241 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:47.762 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:47.763 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:47.763 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:47.764 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-15T02:47:47.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T02:47:47.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T02:47:47.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:47.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-15T02:47:47.873 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:47.873 INFO:tasks.ceph:role: osd.7 2024-09-15T02:47:47.873 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:47:47.874 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T02:47:48.432 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:48.432 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:48.432 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:48.432 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2024-09-15T02:47:48.432 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:48.433 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:48.433 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:48.433 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:48.433 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:48.433 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:48.434 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-15T02:47:48.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-15T02:47:48.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-15T02:47:48.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:48.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-15T02:47:48.488 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:48.488 INFO:tasks.ceph:role: osd.11 2024-09-15T02:47:48.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:47:48.488 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T02:47:48.947 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:48.947 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:48.947 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:48.947 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2024-09-15T02:47:48.947 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:48.948 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:48.948 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:48.948 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:48.948 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:48.948 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:48.949 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-15T02:47:48.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-15T02:47:49.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-15T02:47:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:49.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:49.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-15T02:47:49.033 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T02:47:49.033 INFO:tasks.ceph:role: osd.15 2024-09-15T02:47:49.033 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:47:49.033 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T02:47:49.593 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T02:47:49.593 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T02:47:49.593 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T02:47:49.593 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2024-09-15T02:47:49.593 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T02:47:49.593 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-15T02:47:49.594 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T02:47:49.594 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T02:47:49.594 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T02:47:49.594 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T02:47:49.595 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-15T02:47:49.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-15T02:47:49.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-15T02:47:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:47:49.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T02:47:49.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:49.747 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:49.742+0000 7fc02052d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T02:47:49.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:49.746+0000 7fc02052d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T02:47:49.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:49.746+0000 7fc02052d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T02:47:49.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:49.746+0000 7fc02052d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T02:47:49.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:49.746+0000 7fc02052d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T02:47:49.756 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:49.750+0000 7fc02052d540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T02:47:50.315 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:50.310+0000 7fc02052d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:51.094 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:51.090+0000 7fc02052d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:51.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:51.869+0000 7fc02052d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:52.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T02:47:52.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:52.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:52.773+0000 7ff37147e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-15T02:47:52.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:52.773+0000 7ff37147e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-15T02:47:52.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:52.773+0000 7ff37147e540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T02:47:52.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:52.773+0000 7ff37147e540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T02:47:52.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:52.773+0000 7ff37147e540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T02:47:52.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:52.773+0000 7ff37147e540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-15T02:47:53.335 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:53.329+0000 7ff37147e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:54.121 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:54.113+0000 7ff37147e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:54.903 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:54.897+0000 7ff37147e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:55.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-15T02:47:55.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:55.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:55.797+0000 7fa828a86540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-15T02:47:55.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:55.797+0000 7fa828a86540 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-15T02:47:55.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:55.797+0000 7fa828a86540 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-09-15T02:47:55.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:55.797+0000 7fa828a86540 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-09-15T02:47:55.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:55.797+0000 7fa828a86540 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-09-15T02:47:55.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:55.801+0000 7fa828a86540 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-15T02:47:56.371 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:56.365+0000 7fa828a86540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:57.154 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:57.149+0000 7fa828a86540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:57.941 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:57.937+0000 7fa828a86540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:47:58.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-15T02:47:58.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:47:58.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:58.889+0000 7feb54ae0540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-15T02:47:58.897 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:58.889+0000 7feb54ae0540 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-15T02:47:58.897 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:58.889+0000 7feb54ae0540 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-09-15T02:47:58.897 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:58.889+0000 7feb54ae0540 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-09-15T02:47:58.897 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:58.889+0000 7feb54ae0540 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-09-15T02:47:58.897 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:58.889+0000 7feb54ae0540 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-15T02:47:59.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:47:59.453+0000 7feb54ae0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:48:00.243 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:48:00.237+0000 7feb54ae0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:48:01.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:48:01.025+0000 7feb54ae0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T02:48:01.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-15T02:48:01.896 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T02:48:01.896 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:01.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-15T02:48:01.913 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:01.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T02:48:01.929 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:01.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T02:48:01.940 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:01.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-15T02:48:01.952 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:01.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-15T02:48:02.007 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:02.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T02:48:02.023 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:02.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T02:48:02.034 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:02.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-15T02:48:02.092 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:02.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-15T02:48:02.102 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:02.102 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T02:48:02.119 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:02.119 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-15T02:48:02.177 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:02.177 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-15T02:48:02.186 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:02.187 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-15T02:48:02.200 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:02.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T02:48:02.217 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:02.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T02:48:02.232 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:02.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-15T02:48:02.242 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:02.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-15T02:48:02.300 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:02.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-15T02:48:02.310 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:02.310 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T02:48:02.315 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T02:48:02.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T02:48:02.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T02:48:02.320 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T02:48:02.330 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-09-15T02:48:02.330 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCoSuZmeOb5MBAAHObQlRQfW+u0AorsSdnEaQ== 2024-09-15T02:48:02.330 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-09-15T02:48:02.330 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCrSuZm2R/JFxAAgADHwzfnW7Ojj/3yc8IzEQ== 2024-09-15T02:48:02.330 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCuSuZmwmMwGRAAoZbFdZ2wGdVrnH4rXXcR0g== 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.8] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCxSuZmXTU2HRAAZN/HNQVHxBOnweDajbwcQQ== 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.12] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC0SuZmzlAEIBAAHj169OEZBUkR/rheJg9Pzw== 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC4SuZm9+VwChAAgGzVHnpbOC7novlbi+RIfA== 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC7SuZmjJbLDBAAncs7dR/yTbHrJ//CIv4z8g== 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.9] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC+SuZmQaJiERAAReBUpnAoPptflUo0QcPLPw== 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.13] 2024-09-15T02:48:02.331 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDBSuZm+85/ExAARe/OYGDxyxVpFVxliSjAXQ== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDHSuZmBzCXAhAAHW67YHNn/q0lka3jb8S1BA== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDKSuZmzwYuBRAAzFFAyiJoOvwaH0p7+0gdig== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.10] 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDNSuZmVXG8CRAAM+rDP0KjN9zKFcd1fzus2Q== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.14] 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDQSuZm1cW4DRAA4ArHuL0/zEWefyoxy6Fc2w== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCoSuZmj2fhMxAAu5m17QpWGwvab5l1l4Z0Yw== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDVSuZmPGJ8LBAA1e0EjLg9WE3t1H6da/Jo6Q== 2024-09-15T02:48:02.332 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDYSuZmSbR1LhAAaQRCRhX0aE27i/VZsWZ0bg== 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.11] 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDbSuZmAKTrLxAAMnqH0zlM6JA8ukz/C6jFOw== 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.15] 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDeSuZm3N9TNRAAhM5ZjK6i482VioiQhOyhKg== 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-09-15T02:48:02.333 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCoSuZm5dcENhAAs7vJZnMzJeW6PPm8thZnXg== 2024-09-15T02:48:02.335 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.y] 2024-09-15T02:48:02.335 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCoSuZmeOb5MBAAHObQlRQfW+u0AorsSdnEaQ== 2024-09-15T02:48:02.335 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-09-15T02:48:02.335 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCrSuZm2R/JFxAAgADHwzfnW7Ojj/3yc8IzEQ== 2024-09-15T02:48:02.335 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.4] 2024-09-15T02:48:02.335 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCuSuZmwmMwGRAAoZbFdZ2wGdVrnH4rXXcR0g== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.8] 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCxSuZmXTU2HRAAZN/HNQVHxBOnweDajbwcQQ== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.12] 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC0SuZmzlAEIBAAHj169OEZBUkR/rheJg9Pzw== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC4SuZm9+VwChAAgGzVHnpbOC7novlbi+RIfA== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.5] 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC7SuZmjJbLDBAAncs7dR/yTbHrJ//CIv4z8g== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.9] 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC+SuZmQaJiERAAReBUpnAoPptflUo0QcPLPw== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.13] 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDBSuZm+85/ExAARe/OYGDxyxVpFVxliSjAXQ== 2024-09-15T02:48:02.336 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDHSuZmBzCXAhAAHW67YHNn/q0lka3jb8S1BA== 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.6] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDKSuZmzwYuBRAAzFFAyiJoOvwaH0p7+0gdig== 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.10] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDNSuZmVXG8CRAAM+rDP0KjN9zKFcd1fzus2Q== 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.14] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDQSuZm1cW4DRAA4ArHuL0/zEWefyoxy6Fc2w== 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCoSuZmj2fhMxAAu5m17QpWGwvab5l1l4Z0Yw== 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDVSuZmPGJ8LBAA1e0EjLg9WE3t1H6da/Jo6Q== 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.7] 2024-09-15T02:48:02.337 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDYSuZmSbR1LhAAaQRCRhX0aE27i/VZsWZ0bg== 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.11] 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDbSuZmAKTrLxAAMnqH0zlM6JA8ukz/C6jFOw== 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.15] 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDeSuZm3N9TNRAAhM5ZjK6i482VioiQhOyhKg== 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi152.stdout:[client.0] 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCoSuZm5dcENhAAs7vJZnMzJeW6PPm8thZnXg== 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.y] 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCoSuZmeOb5MBAAHObQlRQfW+u0AorsSdnEaQ== 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCrSuZm2R/JFxAAgADHwzfnW7Ojj/3yc8IzEQ== 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2024-09-15T02:48:02.338 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCuSuZmwmMwGRAAoZbFdZ2wGdVrnH4rXXcR0g== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.8] 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCxSuZmXTU2HRAAZN/HNQVHxBOnweDajbwcQQ== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.12] 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC0SuZmzlAEIBAAHj169OEZBUkR/rheJg9Pzw== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC4SuZm9+VwChAAgGzVHnpbOC7novlbi+RIfA== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC7SuZmjJbLDBAAncs7dR/yTbHrJ//CIv4z8g== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.9] 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC+SuZmQaJiERAAReBUpnAoPptflUo0QcPLPw== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.13] 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDBSuZm+85/ExAARe/OYGDxyxVpFVxliSjAXQ== 2024-09-15T02:48:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDHSuZmBzCXAhAAHW67YHNn/q0lka3jb8S1BA== 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.6] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDKSuZmzwYuBRAAzFFAyiJoOvwaH0p7+0gdig== 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.10] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDNSuZmVXG8CRAAM+rDP0KjN9zKFcd1fzus2Q== 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.14] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDQSuZm1cW4DRAA4ArHuL0/zEWefyoxy6Fc2w== 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCoSuZmj2fhMxAAu5m17QpWGwvab5l1l4Z0Yw== 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDVSuZmPGJ8LBAA1e0EjLg9WE3t1H6da/Jo6Q== 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.7] 2024-09-15T02:48:02.340 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDYSuZmSbR1LhAAaQRCRhX0aE27i/VZsWZ0bg== 2024-09-15T02:48:02.341 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.11] 2024-09-15T02:48:02.341 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDbSuZmAKTrLxAAMnqH0zlM6JA8ukz/C6jFOw== 2024-09-15T02:48:02.341 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.15] 2024-09-15T02:48:02.341 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDeSuZm3N9TNRAAhM5ZjK6i482VioiQhOyhKg== 2024-09-15T02:48:02.341 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-09-15T02:48:02.341 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCoSuZm5dcENhAAs7vJZnMzJeW6PPm8thZnXg== 2024-09-15T02:48:02.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:48:02.379 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:48:02.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:48:02.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.430 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.480 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.561 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.614 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.660 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.744 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.746 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.792 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.794 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.888 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.934 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:02.977 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.028 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.111 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.197 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:48:03.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:48:03.245 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T02:48:03.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.292 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.379 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.432 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.517 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T02:48:03.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:48:03.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:48:03.602 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T02:48:03.651 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T02:48:03.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T02:48:03.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T02:48:03.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T02:48:03.778 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T02:48:03.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T02:48:03.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T02:48:03.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T02:48:03.873 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T02:48:03.929 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T02:48:03.946 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:48:03.948 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:48:03.951 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T02:48:03.996 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T02:48:04.117 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T02:48:04.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T02:48:04.120 INFO:tasks.ceph.mon.a:Started 2024-09-15T02:48:04.120 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T02:48:04.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T02:48:04.123 INFO:tasks.ceph.mon.b:Started 2024-09-15T02:48:04.123 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T02:48:04.123 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T02:48:04.125 INFO:tasks.ceph.mon.c:Started 2024-09-15T02:48:04.125 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T02:48:04.126 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-15T02:48:04.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-15T02:48:04.128 INFO:tasks.ceph.mgr.y:Started 2024-09-15T02:48:04.128 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T02:48:04.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T02:48:04.130 INFO:tasks.ceph.mgr.x:Started 2024-09-15T02:48:04.130 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T02:48:04.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T02:48:10.718 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:10.713+0000 7f685cb28000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T02:48:10.730 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:10.726+0000 7fb86fd25000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T02:48:10.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:10.734+0000 7fea6e59c700 0 --2- 172.21.15.6:0/4045697522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea58003a60 0x7fea58003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T02:48:10.840 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-09-15T02:48:10.852 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T02:48:10.853 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:10.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T02:48:10.869 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:10.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T02:48:10.877 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:10.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-15T02:48:10.887 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-15T02:48:10.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-15T02:48:10.944 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:10.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T02:48:10.961 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:10.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T02:48:10.977 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:10.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-15T02:48:11.035 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-15T02:48:11.035 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-15T02:48:11.047 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:11.047 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T02:48:11.062 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:11.062 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-15T02:48:11.120 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:11.120 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-15T02:48:11.131 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-15T02:48:11.131 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-15T02:48:11.141 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:11.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T02:48:11.157 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:11.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-15T02:48:11.217 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:11.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-15T02:48:11.227 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-15T02:48:11.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-15T02:48:11.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 2d28679c-afcb-472a-8a92-b30cbeb553a7 0 2024-09-15T02:48:11.477 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:11.474+0000 7fb86fd25000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T02:48:11.490 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:11.485+0000 7f685cb28000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T02:48:11.495 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-09-15T02:48:11.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new a3111517-1d9f-4c6b-a838-8eb5e07f8c39 1 2024-09-15T02:48:11.538 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:11.534+0000 7fb86fd25000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T02:48:11.544 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:11.537+0000 7f685cb28000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T02:48:11.655 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:11.649+0000 7f685cb28000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:48:11.659 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:11.654+0000 7fb86fd25000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:48:11.709 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-09-15T02:48:11.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 5a893db6-a9ac-4c69-ac36-57826eab4e3d 2 2024-09-15T02:48:11.879 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:11.873+0000 7f685cb28000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T02:48:11.905 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:11.898+0000 7fb86fd25000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T02:48:11.923 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-09-15T02:48:11.927 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:11.921+0000 7f685cb28000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T02:48:11.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new ab0d2e8a-aedd-4d8b-a5d4-881317345d27 3 2024-09-15T02:48:11.961 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:11.954+0000 7fb86fd25000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T02:48:12.109 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:12.101+0000 7f685cb28000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T02:48:12.142 INFO:teuthology.orchestra.run.smithi191.stdout:3 2024-09-15T02:48:12.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 3edf8310-01a7-4377-a469-b376971149ea 4 2024-09-15T02:48:12.155 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:12.150+0000 7fb86fd25000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T02:48:12.173 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:12.169+0000 7f685cb28000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T02:48:12.224 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:12.218+0000 7fb86fd25000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T02:48:12.236 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:12.229+0000 7f685cb28000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T02:48:12.293 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:12.286+0000 7fb86fd25000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T02:48:12.358 INFO:teuthology.orchestra.run.smithi191.stdout:4 2024-09-15T02:48:12.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 8000b2e4-ca30-4591-88f2-9b0eee27cbca 5 2024-09-15T02:48:12.578 INFO:teuthology.orchestra.run.smithi191.stdout:5 2024-09-15T02:48:12.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 7f8ecd2a-7829-496b-b349-5fdebbb6962a 6 2024-09-15T02:48:12.660 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:12.653+0000 7f685cb28000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T02:48:12.735 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:12.730+0000 7fb86fd25000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T02:48:12.797 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:12.789+0000 7f685cb28000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T02:48:12.823 INFO:teuthology.orchestra.run.smithi191.stdout:6 2024-09-15T02:48:12.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new d94d0835-cb78-46f5-ae93-1b6d60a9ad84 7 2024-09-15T02:48:12.862 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:12.858+0000 7fb86fd25000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T02:48:13.119 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.113+0000 7f685cb28000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T02:48:13.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-15T02:48:13.141+0000 7f09caffd700 0 --2- 172.21.15.191:0/3704657325 >> v2:172.21.15.6:3300/0 conn(0x7f09cc1110c0 0x7f09cc111490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T02:48:13.191 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.186+0000 7fb86fd25000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T02:48:13.253 INFO:teuthology.orchestra.run.smithi191.stdout:7 2024-09-15T02:48:13.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 61e692a5-d458-4bbc-8678-66f2c7b6bfe2 8 2024-09-15T02:48:13.298 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.293+0000 7f685cb28000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:48:13.370 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.366+0000 7fb86fd25000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T02:48:13.372 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.365+0000 7f685cb28000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T02:48:13.430 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.425+0000 7f685cb28000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T02:48:13.441 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.434+0000 7fb86fd25000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T02:48:13.471 INFO:teuthology.orchestra.run.smithi191.stdout:8 2024-09-15T02:48:13.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new b9a7b347-f334-4ccc-82ad-d6d67b366485 9 2024-09-15T02:48:13.492 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.485+0000 7f685cb28000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T02:48:13.496 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.490+0000 7fb86fd25000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T02:48:13.556 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.550+0000 7fb86fd25000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T02:48:13.657 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.649+0000 7f685cb28000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T02:48:13.694 INFO:teuthology.orchestra.run.smithi191.stdout:9 2024-09-15T02:48:13.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new f446f37d-a5c3-4e59-88f9-5a0bb818b35b 10 2024-09-15T02:48:13.716 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.710+0000 7fb86fd25000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T02:48:13.839 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.833+0000 7f685cb28000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T02:48:13.884 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.878+0000 7fb86fd25000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T02:48:13.920 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:13.913+0000 7f685cb28000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T02:48:13.931 INFO:teuthology.orchestra.run.smithi191.stdout:10 2024-09-15T02:48:13.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 78916195-e38b-4ae2-a160-6feb89ec76ff 11 2024-09-15T02:48:13.954 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:13.950+0000 7fb86fd25000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T02:48:14.101 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:14.093+0000 7f685cb28000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T02:48:14.122 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:14.118+0000 7fb86fd25000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T02:48:14.149 INFO:teuthology.orchestra.run.smithi191.stdout:11 2024-09-15T02:48:14.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 778d9a09-0481-4af0-b838-ceeabc6c0af2 12 2024-09-15T02:48:14.175 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:14.169+0000 7f685cb28000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T02:48:14.196 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:14.190+0000 7fb86fd25000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T02:48:14.361 INFO:teuthology.orchestra.run.smithi191.stdout:12 2024-09-15T02:48:14.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 5167c2e2-7879-4a97-9966-f0621d9ced72 13 2024-09-15T02:48:14.621 INFO:teuthology.orchestra.run.smithi191.stdout:13 2024-09-15T02:48:14.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 7b4d6903-aa33-4280-89f1-05c499290d8f 14 2024-09-15T02:48:14.839 INFO:teuthology.orchestra.run.smithi191.stdout:14 2024-09-15T02:48:14.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new baf2647e-ce90-42a7-82ae-ec7578ef31da 15 2024-09-15T02:48:14.911 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:14.905+0000 7f685cb28000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T02:48:14.938 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-09-15T02:48:14.934+0000 7fb86fd25000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T02:48:15.050 INFO:teuthology.orchestra.run.smithi191.stdout:15 2024-09-15T02:48:15.067 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T02:48:15.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T02:48:15.070 INFO:tasks.ceph.osd.0:Started 2024-09-15T02:48:15.070 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T02:48:15.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T02:48:15.072 INFO:tasks.ceph.osd.4:Started 2024-09-15T02:48:15.072 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-15T02:48:15.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-15T02:48:15.078 INFO:tasks.ceph.osd.8:Started 2024-09-15T02:48:15.078 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-15T02:48:15.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-15T02:48:15.081 INFO:tasks.ceph.osd.12:Started 2024-09-15T02:48:15.081 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T02:48:15.081 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T02:48:15.083 INFO:tasks.ceph.osd.1:Started 2024-09-15T02:48:15.083 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T02:48:15.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T02:48:15.086 INFO:tasks.ceph.osd.5:Started 2024-09-15T02:48:15.086 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-15T02:48:15.086 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-15T02:48:15.088 INFO:tasks.ceph.osd.9:Started 2024-09-15T02:48:15.088 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-15T02:48:15.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-15T02:48:15.091 INFO:tasks.ceph.osd.13:Started 2024-09-15T02:48:15.091 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T02:48:15.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T02:48:15.094 INFO:tasks.ceph.osd.2:Started 2024-09-15T02:48:15.094 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-15T02:48:15.094 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-15T02:48:15.096 INFO:tasks.ceph.osd.6:Started 2024-09-15T02:48:15.096 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-15T02:48:15.096 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-15T02:48:15.099 INFO:tasks.ceph.osd.10:Started 2024-09-15T02:48:15.099 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-15T02:48:15.099 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-15T02:48:15.101 INFO:tasks.ceph.osd.14:Started 2024-09-15T02:48:15.101 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T02:48:15.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T02:48:15.104 INFO:tasks.ceph.osd.3:Started 2024-09-15T02:48:15.104 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-15T02:48:15.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-15T02:48:15.106 INFO:tasks.ceph.osd.7:Started 2024-09-15T02:48:15.107 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-15T02:48:15.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-15T02:48:15.109 INFO:tasks.ceph.osd.11:Started 2024-09-15T02:48:15.109 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-15T02:48:15.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-15T02:48:15.111 INFO:tasks.ceph.osd.15:Started 2024-09-15T02:48:15.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:48:15.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:15.295 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:48:15.043044+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T02:48:15.043044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:48:15.307 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T02:48:15.307 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T02:48:15.442 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:15.438+0000 7f3beb5b7540 -1 Falling back to public interface 2024-09-15T02:48:15.443 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:15.438+0000 7f7235f3f540 -1 Falling back to public interface 2024-09-15T02:48:15.463 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:15.458+0000 7fe70aea3540 -1 Falling back to public interface 2024-09-15T02:48:15.479 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:15.473+0000 7f5a3e894540 -1 Falling back to public interface 2024-09-15T02:48:15.487 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:15.482+0000 7fe731772540 -1 Falling back to public interface 2024-09-15T02:48:15.675 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:15.670+0000 7f238c192540 -1 Falling back to public interface 2024-09-15T02:48:15.702 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:15.698+0000 7fed82bb1540 -1 Falling back to public interface 2024-09-15T02:48:15.702 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:15.698+0000 7fe719138540 -1 Falling back to public interface 2024-09-15T02:48:15.711 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:15.705+0000 7f41820f8540 -1 Falling back to public interface 2024-09-15T02:48:15.715 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:15.710+0000 7f91ab0fe540 -1 Falling back to public interface 2024-09-15T02:48:15.715 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:15.710+0000 7ff1ad899540 -1 Falling back to public interface 2024-09-15T02:48:15.716 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:15.710+0000 7f2c44287540 -1 Falling back to public interface 2024-09-15T02:48:15.716 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:15.710+0000 7f93d3fb9540 -1 Falling back to public interface 2024-09-15T02:48:15.717 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:15.714+0000 7f0e3df49540 -1 Falling back to public interface 2024-09-15T02:48:15.726 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:15.721+0000 7f53f4bf1540 -1 Falling back to public interface 2024-09-15T02:48:15.727 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:15.721+0000 7f2365063540 -1 Falling back to public interface 2024-09-15T02:48:15.968 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:15.961+0000 7f6845b9c700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T02:48:15.969 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-15T02:48:15.965+0000 7f6845b9c700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T02:48:16.920 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T02:48:17.220 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-15T02:48:17.220 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":18,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:48:15.043044+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T02:48:15.043044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:48:17.232 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-15T02:48:19.216 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:19.210+0000 7f3beb5b7540 -1 osd.5 0 log_to_monitors true 2024-09-15T02:48:19.273 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:19.265+0000 7f5a3e894540 -1 osd.7 0 log_to_monitors true 2024-09-15T02:48:19.315 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:19.306+0000 7ff1ad899540 -1 osd.12 0 log_to_monitors true 2024-09-15T02:48:19.327 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:19.322+0000 7fe70aea3540 -1 osd.6 0 log_to_monitors true 2024-09-15T02:48:19.331 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:19.326+0000 7f7235f3f540 -1 osd.4 0 log_to_monitors true 2024-09-15T02:48:19.335 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:19.330+0000 7fe719138540 -1 osd.9 0 log_to_monitors true 2024-09-15T02:48:19.338 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:19.330+0000 7fe731772540 -1 osd.8 0 log_to_monitors true 2024-09-15T02:48:19.414 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:19.410+0000 7fed82bb1540 -1 osd.1 0 log_to_monitors true 2024-09-15T02:48:19.435 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:19.429+0000 7f2365063540 -1 osd.3 0 log_to_monitors true 2024-09-15T02:48:19.456 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:19.450+0000 7f238c192540 -1 osd.0 0 log_to_monitors true 2024-09-15T02:48:19.467 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:19.462+0000 7f2c44287540 -1 osd.2 0 log_to_monitors true 2024-09-15T02:48:19.493 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:19.485+0000 7f53f4bf1540 -1 osd.11 0 log_to_monitors true 2024-09-15T02:48:19.527 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:19.522+0000 7f0e3df49540 -1 osd.13 0 log_to_monitors true 2024-09-15T02:48:19.563 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:19.557+0000 7f41820f8540 -1 osd.15 0 log_to_monitors true 2024-09-15T02:48:19.568 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:19.562+0000 7f91ab0fe540 -1 osd.14 0 log_to_monitors true 2024-09-15T02:48:19.597 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:19.594+0000 7f93d3fb9540 -1 osd.10 0 log_to_monitors true 2024-09-15T02:48:20.307 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:20.302+0000 7f3be6d57700 -1 osd.5 0 waiting for initial osdmap 2024-09-15T02:48:20.316 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:20.310+0000 7f3be22ff700 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.312 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:21.306+0000 7f7232f04700 -1 osd.4 0 waiting for initial osdmap 2024-09-15T02:48:21.313 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:21.306+0000 7fe706643700 -1 osd.6 0 waiting for initial osdmap 2024-09-15T02:48:21.313 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:21.305+0000 7f5a3a034700 -1 osd.7 0 waiting for initial osdmap 2024-09-15T02:48:21.316 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:21.314+0000 7fed7e351700 -1 osd.1 0 waiting for initial osdmap 2024-09-15T02:48:21.317 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:21.309+0000 7f417f0bd700 -1 osd.15 0 waiting for initial osdmap 2024-09-15T02:48:21.317 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:21.314+0000 7fe7148d8700 -1 osd.9 0 waiting for initial osdmap 2024-09-15T02:48:21.317 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:21.314+0000 7f2387932700 -1 osd.0 0 waiting for initial osdmap 2024-09-15T02:48:21.318 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:21.313+0000 7f53f0391700 -1 osd.11 0 waiting for initial osdmap 2024-09-15T02:48:21.319 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:21.314+0000 7f93cf759700 -1 osd.10 0 waiting for initial osdmap 2024-09-15T02:48:21.322 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:21.318+0000 7f0e396e9700 -1 osd.13 0 waiting for initial osdmap 2024-09-15T02:48:21.323 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:21.318+0000 7ff1a9039700 -1 osd.12 0 waiting for initial osdmap 2024-09-15T02:48:21.323 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:21.318+0000 7fe72cf12700 -1 osd.8 0 waiting for initial osdmap 2024-09-15T02:48:21.323 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:21.318+0000 7f2c3fa27700 -1 osd.2 0 waiting for initial osdmap 2024-09-15T02:48:21.324 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:21.322+0000 7fe70fe80700 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.324 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:21.317+0000 7f5a355dc700 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.324 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:21.317+0000 7f2360803700 -1 osd.3 0 waiting for initial osdmap 2024-09-15T02:48:21.325 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:21.322+0000 7fed790f8700 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.325 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:21.322+0000 7f91a689e700 -1 osd.14 0 waiting for initial osdmap 2024-09-15T02:48:21.326 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:21.321+0000 7f4178e40700 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.327 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:21.322+0000 7f722cc87700 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.328 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:21.322+0000 7fe701beb700 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.329 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:21.326+0000 7f0e34c91700 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.330 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:21.326+0000 7f2382eda700 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.331 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:21.325+0000 7f53eb939700 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.331 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:21.326+0000 7fe7284ba700 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.331 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:21.326+0000 7f2c3afcf700 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.332 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:21.326+0000 7f93cad01700 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.332 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:21.325+0000 7f235bdab700 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.335 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:21.330+0000 7f91a1e46700 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:21.336 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:21.330+0000 7ff1a3de0700 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:24.848 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T02:48:25.150 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-15T02:48:25.151 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":24,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:48:24.330699+0000","last_up_change":"2024-09-15T02:48:22.306891+0000","last_in_change":"2024-09-15T02:48:15.043044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":22076},{"type":"v1","addr":"172.21.15.152:6801","nonce":22076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":22076},{"type":"v1","addr":"172.21.15.152:6803","nonce":22076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":22076},{"type":"v1","addr":"172.21.15.152:6807","nonce":22076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":22076},{"type":"v1","addr":"172.21.15.152:6805","nonce":22076}]},"public_addr":"172.21.15.152:6801/22076","cluster_addr":"172.21.15.152:6803/22076","heartbeat_back_addr":"172.21.15.152:6807/22076","heartbeat_front_addr":"172.21.15.152:6805/22076","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":22075},{"type":"v1","addr":"172.21.15.152:6812","nonce":22075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":22075},{"type":"v1","addr":"172.21.15.152:6819","nonce":22075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":22075},{"type":"v1","addr":"172.21.15.152:6831","nonce":22075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":22075},{"type":"v1","addr":"172.21.15.152:6826","nonce":22075}]},"public_addr":"172.21.15.152:6812/22075","cluster_addr":"172.21.15.152:6819/22075","heartbeat_back_addr":"172.21.15.152:6831/22075","heartbeat_front_addr":"172.21.15.152:6826/22075","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.496434+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.374590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.432280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.446858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.340414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.173357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.315226+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.292854+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.291714+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.339470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.593108+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.537787+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.297229+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.558699+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.578841+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.554724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:48:25.164 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-15T02:48:25.164 INFO:tasks.ceph:Creating RBD pool 2024-09-15T02:48:25.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T02:48:26.400 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-09-15T02:48:26.415 DEBUG:teuthology.orchestra.run.smithi006:> rbd --cluster ceph pool init rbd 2024-09-15T02:48:29.431 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T02:48:29.432 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T02:48:29.432 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T02:48:29.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T02:48:29.433 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T02:48:29.797 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:29.812 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":21903},{"type":"v1","addr":"172.21.15.191:6801","nonce":21903}]},"active_addr":"172.21.15.191:6801/21903","active_change":"2024-09-15T02:48:14.929157+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":3036825489}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":3037228148}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2838920161}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2616888047}]}]} 2024-09-15T02:48:29.818 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T02:48:29.818 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T02:48:29.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:48:30.126 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:30.126 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":29,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:48:29.405812+0000","last_up_change":"2024-09-15T02:48:22.306891+0000","last_in_change":"2024-09-15T02:48:15.043044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":22076},{"type":"v1","addr":"172.21.15.152:6801","nonce":22076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":22076},{"type":"v1","addr":"172.21.15.152:6803","nonce":22076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":22076},{"type":"v1","addr":"172.21.15.152:6807","nonce":22076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":22076},{"type":"v1","addr":"172.21.15.152:6805","nonce":22076}]},"public_addr":"172.21.15.152:6801/22076","cluster_addr":"172.21.15.152:6803/22076","heartbeat_back_addr":"172.21.15.152:6807/22076","heartbeat_front_addr":"172.21.15.152:6805/22076","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":22075},{"type":"v1","addr":"172.21.15.152:6812","nonce":22075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":22075},{"type":"v1","addr":"172.21.15.152:6819","nonce":22075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":22075},{"type":"v1","addr":"172.21.15.152:6831","nonce":22075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":22075},{"type":"v1","addr":"172.21.15.152:6826","nonce":22075}]},"public_addr":"172.21.15.152:6812/22075","cluster_addr":"172.21.15.152:6819/22075","heartbeat_back_addr":"172.21.15.152:6831/22075","heartbeat_front_addr":"172.21.15.152:6826/22075","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.496434+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.374590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.432280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.446858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.340414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.173357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.315226+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.292854+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.291714+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.339470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.593108+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.537787+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.297229+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.558699+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.578841+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.554724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:48:30.140 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T02:48:30.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:48:30.441 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:30.441 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":29,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:48:29.405812+0000","last_up_change":"2024-09-15T02:48:22.306891+0000","last_in_change":"2024-09-15T02:48:15.043044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":22076},{"type":"v1","addr":"172.21.15.152:6801","nonce":22076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":22076},{"type":"v1","addr":"172.21.15.152:6803","nonce":22076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":22076},{"type":"v1","addr":"172.21.15.152:6807","nonce":22076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":22076},{"type":"v1","addr":"172.21.15.152:6805","nonce":22076}]},"public_addr":"172.21.15.152:6801/22076","cluster_addr":"172.21.15.152:6803/22076","heartbeat_back_addr":"172.21.15.152:6807/22076","heartbeat_front_addr":"172.21.15.152:6805/22076","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":22075},{"type":"v1","addr":"172.21.15.152:6812","nonce":22075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":22075},{"type":"v1","addr":"172.21.15.152:6819","nonce":22075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":22075},{"type":"v1","addr":"172.21.15.152:6831","nonce":22075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":22075},{"type":"v1","addr":"172.21.15.152:6826","nonce":22075}]},"public_addr":"172.21.15.152:6812/22075","cluster_addr":"172.21.15.152:6819/22075","heartbeat_back_addr":"172.21.15.152:6831/22075","heartbeat_front_addr":"172.21.15.152:6826/22075","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.496434+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.374590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.432280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.446858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.340414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.173357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.315226+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.292854+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.291714+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.339470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.593108+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.537787+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.297229+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.558699+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.578841+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.554724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:48:30.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T02:48:30.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T02:48:30.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T02:48:30.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T02:48:30.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T02:48:30.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T02:48:30.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T02:48:30.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T02:48:30.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-15T02:48:30.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-15T02:48:30.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-15T02:48:30.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-15T02:48:30.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-15T02:48:30.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-15T02:48:30.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-15T02:48:30.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-15T02:48:30.792 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T02:48:30.810 INFO:teuthology.orchestra.run.smithi006.stdout:90194313220 2024-09-15T02:48:30.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T02:48:30.813 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T02:48:30.818 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-15T02:48:30.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T02:48:30.829 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T02:48:30.834 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T02:48:30.849 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-15T02:48:30.853 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T02:48:30.861 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T02:48:30.862 INFO:teuthology.orchestra.run.smithi006.stdout:94489280517 2024-09-15T02:48:30.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T02:48:30.888 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-15T02:48:30.906 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T02:48:30.909 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-15T02:48:30.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T02:48:30.911 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-15T02:48:30.926 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T02:48:30.928 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:30.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-15T02:48:31.535 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.569 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.0 2024-09-15T02:48:31.570 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.8 2024-09-15T02:48:31.599 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.609 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.2 2024-09-15T02:48:31.639 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.687 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-15T02:48:31.701 INFO:teuthology.orchestra.run.smithi006.stdout:90194313220 2024-09-15T02:48:31.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2024-09-15T02:48:31.723 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.5 2024-09-15T02:48:31.740 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.744 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.744 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.747 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.755 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-15T02:48:31.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.4 2024-09-15T02:48:31.766 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2024-09-15T02:48:31.783 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.1 2024-09-15T02:48:31.784 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.11 2024-09-15T02:48:31.785 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.785 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.799 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.12 2024-09-15T02:48:31.799 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.813 INFO:teuthology.orchestra.run.smithi006.stdout:94489280517 2024-09-15T02:48:31.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.7 2024-09-15T02:48:31.824 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.832 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.15 2024-09-15T02:48:31.849 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.849 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.859 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.861 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.9 2024-09-15T02:48:31.869 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-15T02:48:31.870 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.6 2024-09-15T02:48:31.875 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.13 2024-09-15T02:48:31.875 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.14 2024-09-15T02:48:31.878 DEBUG:teuthology.parallel:result is None 2024-09-15T02:48:31.878 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T02:48:31.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:48:32.216 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:32.219 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:48:32.234 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T02:48:30.949636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4762272,"kb_used_data":4500,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505187168,"statfs":{"total":1546188226560,"available":1541311660032,"internally_reserved":0,"allocated":4608000,"data_stored":3093008,"data_compressed":217605,"data_compressed_allocated":520192,"data_compressed_original":1933312,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.561732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418030+0000","last_change":"2024-09-15T02:48:29.418551+0000","last_active":"2024-09-15T02:48:29.418030+0000","last_peered":"2024-09-15T02:48:29.418030+0000","last_clean":"2024-09-15T02:48:29.418030+0000","last_became_active":"2024-09-15T02:48:28.178543+0000","last_became_peered":"2024-09-15T02:48:28.178543+0000","last_unstale":"2024-09-15T02:48:29.418030+0000","last_undegraded":"2024-09-15T02:48:29.418030+0000","last_fullsized":"2024-09-15T02:48:29.418030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.111072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073976800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.426256+0000","last_change":"2024-09-15T02:48:29.426846+0000","last_active":"2024-09-15T02:48:29.426256+0000","last_peered":"2024-09-15T02:48:29.426256+0000","last_clean":"2024-09-15T02:48:29.426256+0000","last_became_active":"2024-09-15T02:48:27.400822+0000","last_became_peered":"2024-09-15T02:48:27.400822+0000","last_unstale":"2024-09-15T02:48:29.426256+0000","last_undegraded":"2024-09-15T02:48:29.426256+0000","last_fullsized":"2024-09-15T02:48:29.426256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:32.325018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.419209+0000","last_change":"2024-09-15T02:48:29.419531+0000","last_active":"2024-09-15T02:48:29.419209+0000","last_peered":"2024-09-15T02:48:29.419209+0000","last_clean":"2024-09-15T02:48:29.419209+0000","last_became_active":"2024-09-15T02:48:28.248628+0000","last_became_peered":"2024-09-15T02:48:28.248628+0000","last_unstale":"2024-09-15T02:48:29.419209+0000","last_undegraded":"2024-09-15T02:48:29.419209+0000","last_fullsized":"2024-09-15T02:48:29.419209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:53.626378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055936099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.823459+0000","last_change":"2024-09-15T02:48:29.823846+0000","last_active":"2024-09-15T02:48:29.823459+0000","last_peered":"2024-09-15T02:48:29.823459+0000","last_clean":"2024-09-15T02:48:29.823459+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:48:29.823459+0000","last_undegraded":"2024-09-15T02:48:29.823459+0000","last_fullsized":"2024-09-15T02:48:29.823459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:26.621594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418653+0000","last_change":"2024-09-15T02:48:29.418653+0000","last_active":"2024-09-15T02:48:29.418653+0000","last_peered":"2024-09-15T02:48:29.418653+0000","last_clean":"2024-09-15T02:48:29.418653+0000","last_became_active":"2024-09-15T02:48:27.398299+0000","last_became_peered":"2024-09-15T02:48:27.398299+0000","last_unstale":"2024-09-15T02:48:29.418653+0000","last_undegraded":"2024-09-15T02:48:29.418653+0000","last_fullsized":"2024-09-15T02:48:29.418653+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.751755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.420449+0000","last_change":"2024-09-15T02:48:29.420976+0000","last_active":"2024-09-15T02:48:29.420449+0000","last_peered":"2024-09-15T02:48:29.420449+0000","last_clean":"2024-09-15T02:48:29.420449+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:48:29.420449+0000","last_undegraded":"2024-09-15T02:48:29.420449+0000","last_fullsized":"2024-09-15T02:48:29.420449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:33.874284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418630+0000","last_change":"2024-09-15T02:48:29.418940+0000","last_active":"2024-09-15T02:48:29.418630+0000","last_peered":"2024-09-15T02:48:29.418630+0000","last_clean":"2024-09-15T02:48:29.418630+0000","last_became_active":"2024-09-15T02:48:27.395014+0000","last_became_peered":"2024-09-15T02:48:27.395014+0000","last_unstale":"2024-09-15T02:48:29.418630+0000","last_undegraded":"2024-09-15T02:48:29.418630+0000","last_fullsized":"2024-09-15T02:48:29.418630+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:45.306020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011443880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.636075+0000","last_change":"2024-09-15T02:48:29.636494+0000","last_active":"2024-09-15T02:48:29.636075+0000","last_peered":"2024-09-15T02:48:29.636075+0000","last_clean":"2024-09-15T02:48:29.636075+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:48:29.636075+0000","last_undegraded":"2024-09-15T02:48:29.636075+0000","last_fullsized":"2024-09-15T02:48:29.636075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:30.837572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T02:48:28.637141+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:48:28.637141+0000","last_peered":"2024-09-15T02:48:28.637141+0000","last_clean":"2024-09-15T02:48:28.637141+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:48:28.637141+0000","last_undegraded":"2024-09-15T02:48:28.637141+0000","last_fullsized":"2024-09-15T02:48:28.637141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.005063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.415}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.278}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6989999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.577}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5139999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0779999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]}]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9550000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.363}]}]},{"osd":0,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":266240,"data_stored":117823,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.992}]}]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.504}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.133}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.468}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0390000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]}]},{"osd":15,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,4,5,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0329999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1709999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9630000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297640,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074200,"statfs":{"total":96636764160,"available":96331980800,"internally_reserved":0,"allocated":286720,"data_stored":127954,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.331}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4279999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0539999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.101}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":423.70800000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4710000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6659999999999999}]}]},{"osd":9,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127376,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.198}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.978}]}]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7829999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.724}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5739999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9530000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6680000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:48:32.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:48:32.533 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:48:32.547 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T02:48:30.949636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4762272,"kb_used_data":4500,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505187168,"statfs":{"total":1546188226560,"available":1541311660032,"internally_reserved":0,"allocated":4608000,"data_stored":3093008,"data_compressed":217605,"data_compressed_allocated":520192,"data_compressed_original":1933312,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.561732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418030+0000","last_change":"2024-09-15T02:48:29.418551+0000","last_active":"2024-09-15T02:48:29.418030+0000","last_peered":"2024-09-15T02:48:29.418030+0000","last_clean":"2024-09-15T02:48:29.418030+0000","last_became_active":"2024-09-15T02:48:28.178543+0000","last_became_peered":"2024-09-15T02:48:28.178543+0000","last_unstale":"2024-09-15T02:48:29.418030+0000","last_undegraded":"2024-09-15T02:48:29.418030+0000","last_fullsized":"2024-09-15T02:48:29.418030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.111072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073976800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.426256+0000","last_change":"2024-09-15T02:48:29.426846+0000","last_active":"2024-09-15T02:48:29.426256+0000","last_peered":"2024-09-15T02:48:29.426256+0000","last_clean":"2024-09-15T02:48:29.426256+0000","last_became_active":"2024-09-15T02:48:27.400822+0000","last_became_peered":"2024-09-15T02:48:27.400822+0000","last_unstale":"2024-09-15T02:48:29.426256+0000","last_undegraded":"2024-09-15T02:48:29.426256+0000","last_fullsized":"2024-09-15T02:48:29.426256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:32.325018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.419209+0000","last_change":"2024-09-15T02:48:29.419531+0000","last_active":"2024-09-15T02:48:29.419209+0000","last_peered":"2024-09-15T02:48:29.419209+0000","last_clean":"2024-09-15T02:48:29.419209+0000","last_became_active":"2024-09-15T02:48:28.248628+0000","last_became_peered":"2024-09-15T02:48:28.248628+0000","last_unstale":"2024-09-15T02:48:29.419209+0000","last_undegraded":"2024-09-15T02:48:29.419209+0000","last_fullsized":"2024-09-15T02:48:29.419209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:53.626378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055936099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.823459+0000","last_change":"2024-09-15T02:48:29.823846+0000","last_active":"2024-09-15T02:48:29.823459+0000","last_peered":"2024-09-15T02:48:29.823459+0000","last_clean":"2024-09-15T02:48:29.823459+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:48:29.823459+0000","last_undegraded":"2024-09-15T02:48:29.823459+0000","last_fullsized":"2024-09-15T02:48:29.823459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:26.621594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418653+0000","last_change":"2024-09-15T02:48:29.418653+0000","last_active":"2024-09-15T02:48:29.418653+0000","last_peered":"2024-09-15T02:48:29.418653+0000","last_clean":"2024-09-15T02:48:29.418653+0000","last_became_active":"2024-09-15T02:48:27.398299+0000","last_became_peered":"2024-09-15T02:48:27.398299+0000","last_unstale":"2024-09-15T02:48:29.418653+0000","last_undegraded":"2024-09-15T02:48:29.418653+0000","last_fullsized":"2024-09-15T02:48:29.418653+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.751755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.420449+0000","last_change":"2024-09-15T02:48:29.420976+0000","last_active":"2024-09-15T02:48:29.420449+0000","last_peered":"2024-09-15T02:48:29.420449+0000","last_clean":"2024-09-15T02:48:29.420449+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:48:29.420449+0000","last_undegraded":"2024-09-15T02:48:29.420449+0000","last_fullsized":"2024-09-15T02:48:29.420449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:33.874284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418630+0000","last_change":"2024-09-15T02:48:29.418940+0000","last_active":"2024-09-15T02:48:29.418630+0000","last_peered":"2024-09-15T02:48:29.418630+0000","last_clean":"2024-09-15T02:48:29.418630+0000","last_became_active":"2024-09-15T02:48:27.395014+0000","last_became_peered":"2024-09-15T02:48:27.395014+0000","last_unstale":"2024-09-15T02:48:29.418630+0000","last_undegraded":"2024-09-15T02:48:29.418630+0000","last_fullsized":"2024-09-15T02:48:29.418630+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:45.306020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011443880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.636075+0000","last_change":"2024-09-15T02:48:29.636494+0000","last_active":"2024-09-15T02:48:29.636075+0000","last_peered":"2024-09-15T02:48:29.636075+0000","last_clean":"2024-09-15T02:48:29.636075+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:48:29.636075+0000","last_undegraded":"2024-09-15T02:48:29.636075+0000","last_fullsized":"2024-09-15T02:48:29.636075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:30.837572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T02:48:28.637141+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:48:28.637141+0000","last_peered":"2024-09-15T02:48:28.637141+0000","last_clean":"2024-09-15T02:48:28.637141+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:48:28.637141+0000","last_undegraded":"2024-09-15T02:48:28.637141+0000","last_fullsized":"2024-09-15T02:48:28.637141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.005063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.415}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.278}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6989999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.577}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5139999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0779999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]}]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9550000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.363}]}]},{"osd":0,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":266240,"data_stored":117823,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.992}]}]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.504}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.133}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.468}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0390000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]}]},{"osd":15,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,4,5,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0329999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1709999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9630000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297640,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074200,"statfs":{"total":96636764160,"available":96331980800,"internally_reserved":0,"allocated":286720,"data_stored":127954,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.331}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4279999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0539999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.101}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":423.70800000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4710000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6659999999999999}]}]},{"osd":9,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127376,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.198}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.978}]}]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7829999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.724}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5739999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9530000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6680000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:48:32.549 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T02:48:32.549 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T02:48:32.550 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T02:48:32.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T02:48:32.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:32.866 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T02:48:32.879 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T02:48:32.879 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-15T02:48:32.890 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-15T02:48:32.890 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:48:32.891 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-15T02:48:32.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-15T02:48:33.217 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-15T02:48:33.217 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T02:48:33.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:48:33.522 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:33.523 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:48:33.537 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-15T02:48:32.950324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4762288,"kb_used_data":4512,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505187152,"statfs":{"total":1546188226560,"available":1541311643648,"internally_reserved":0,"allocated":4620288,"data_stored":3102542,"data_compressed":219450,"data_compressed_allocated":524288,"data_compressed_original":1941504,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.562420"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418030+0000","last_change":"2024-09-15T02:48:29.418551+0000","last_active":"2024-09-15T02:48:29.418030+0000","last_peered":"2024-09-15T02:48:29.418030+0000","last_clean":"2024-09-15T02:48:29.418030+0000","last_became_active":"2024-09-15T02:48:28.178543+0000","last_became_peered":"2024-09-15T02:48:28.178543+0000","last_unstale":"2024-09-15T02:48:29.418030+0000","last_undegraded":"2024-09-15T02:48:29.418030+0000","last_fullsized":"2024-09-15T02:48:29.418030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.111072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073976800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.426256+0000","last_change":"2024-09-15T02:48:29.426846+0000","last_active":"2024-09-15T02:48:29.426256+0000","last_peered":"2024-09-15T02:48:29.426256+0000","last_clean":"2024-09-15T02:48:29.426256+0000","last_became_active":"2024-09-15T02:48:27.400822+0000","last_became_peered":"2024-09-15T02:48:27.400822+0000","last_unstale":"2024-09-15T02:48:29.426256+0000","last_undegraded":"2024-09-15T02:48:29.426256+0000","last_fullsized":"2024-09-15T02:48:29.426256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:32.325018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.419209+0000","last_change":"2024-09-15T02:48:29.419531+0000","last_active":"2024-09-15T02:48:29.419209+0000","last_peered":"2024-09-15T02:48:29.419209+0000","last_clean":"2024-09-15T02:48:29.419209+0000","last_became_active":"2024-09-15T02:48:28.248628+0000","last_became_peered":"2024-09-15T02:48:28.248628+0000","last_unstale":"2024-09-15T02:48:29.419209+0000","last_undegraded":"2024-09-15T02:48:29.419209+0000","last_fullsized":"2024-09-15T02:48:29.419209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:53.626378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055936099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.823459+0000","last_change":"2024-09-15T02:48:29.823846+0000","last_active":"2024-09-15T02:48:29.823459+0000","last_peered":"2024-09-15T02:48:29.823459+0000","last_clean":"2024-09-15T02:48:29.823459+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:48:29.823459+0000","last_undegraded":"2024-09-15T02:48:29.823459+0000","last_fullsized":"2024-09-15T02:48:29.823459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:26.621594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418653+0000","last_change":"2024-09-15T02:48:29.418653+0000","last_active":"2024-09-15T02:48:29.418653+0000","last_peered":"2024-09-15T02:48:29.418653+0000","last_clean":"2024-09-15T02:48:29.418653+0000","last_became_active":"2024-09-15T02:48:27.398299+0000","last_became_peered":"2024-09-15T02:48:27.398299+0000","last_unstale":"2024-09-15T02:48:29.418653+0000","last_undegraded":"2024-09-15T02:48:29.418653+0000","last_fullsized":"2024-09-15T02:48:29.418653+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.751755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.420449+0000","last_change":"2024-09-15T02:48:29.420976+0000","last_active":"2024-09-15T02:48:29.420449+0000","last_peered":"2024-09-15T02:48:29.420449+0000","last_clean":"2024-09-15T02:48:29.420449+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:48:29.420449+0000","last_undegraded":"2024-09-15T02:48:29.420449+0000","last_fullsized":"2024-09-15T02:48:29.420449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:33.874284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418630+0000","last_change":"2024-09-15T02:48:29.418940+0000","last_active":"2024-09-15T02:48:29.418630+0000","last_peered":"2024-09-15T02:48:29.418630+0000","last_clean":"2024-09-15T02:48:29.418630+0000","last_became_active":"2024-09-15T02:48:27.395014+0000","last_became_peered":"2024-09-15T02:48:27.395014+0000","last_unstale":"2024-09-15T02:48:29.418630+0000","last_undegraded":"2024-09-15T02:48:29.418630+0000","last_fullsized":"2024-09-15T02:48:29.418630+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:45.306020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011443880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.636075+0000","last_change":"2024-09-15T02:48:29.636494+0000","last_active":"2024-09-15T02:48:29.636075+0000","last_peered":"2024-09-15T02:48:29.636075+0000","last_clean":"2024-09-15T02:48:29.636075+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:48:29.636075+0000","last_undegraded":"2024-09-15T02:48:29.636075+0000","last_fullsized":"2024-09-15T02:48:29.636075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:30.837572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T02:48:28.637141+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:48:28.637141+0000","last_peered":"2024-09-15T02:48:28.637141+0000","last_clean":"2024-09-15T02:48:28.637141+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:48:28.637141+0000","last_undegraded":"2024-09-15T02:48:28.637141+0000","last_fullsized":"2024-09-15T02:48:28.637141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.005063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.415}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.278}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6989999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.577}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5139999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0779999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7070000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.149}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9470000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.992}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]}]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,4,5,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0329999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1709999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9630000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297640,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074200,"statfs":{"total":96636764160,"available":96331980800,"internally_reserved":0,"allocated":286720,"data_stored":127954,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":640.51300000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.101}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127376,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.198}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.978}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7829999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:48:33.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:48:33.844 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:48:33.847 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:48:33.861 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-15T02:48:32.950324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4762288,"kb_used_data":4512,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505187152,"statfs":{"total":1546188226560,"available":1541311643648,"internally_reserved":0,"allocated":4620288,"data_stored":3102542,"data_compressed":219450,"data_compressed_allocated":524288,"data_compressed_original":1941504,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.562420"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418030+0000","last_change":"2024-09-15T02:48:29.418551+0000","last_active":"2024-09-15T02:48:29.418030+0000","last_peered":"2024-09-15T02:48:29.418030+0000","last_clean":"2024-09-15T02:48:29.418030+0000","last_became_active":"2024-09-15T02:48:28.178543+0000","last_became_peered":"2024-09-15T02:48:28.178543+0000","last_unstale":"2024-09-15T02:48:29.418030+0000","last_undegraded":"2024-09-15T02:48:29.418030+0000","last_fullsized":"2024-09-15T02:48:29.418030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.111072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073976800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.426256+0000","last_change":"2024-09-15T02:48:29.426846+0000","last_active":"2024-09-15T02:48:29.426256+0000","last_peered":"2024-09-15T02:48:29.426256+0000","last_clean":"2024-09-15T02:48:29.426256+0000","last_became_active":"2024-09-15T02:48:27.400822+0000","last_became_peered":"2024-09-15T02:48:27.400822+0000","last_unstale":"2024-09-15T02:48:29.426256+0000","last_undegraded":"2024-09-15T02:48:29.426256+0000","last_fullsized":"2024-09-15T02:48:29.426256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:32.325018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.419209+0000","last_change":"2024-09-15T02:48:29.419531+0000","last_active":"2024-09-15T02:48:29.419209+0000","last_peered":"2024-09-15T02:48:29.419209+0000","last_clean":"2024-09-15T02:48:29.419209+0000","last_became_active":"2024-09-15T02:48:28.248628+0000","last_became_peered":"2024-09-15T02:48:28.248628+0000","last_unstale":"2024-09-15T02:48:29.419209+0000","last_undegraded":"2024-09-15T02:48:29.419209+0000","last_fullsized":"2024-09-15T02:48:29.419209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:53.626378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055936099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.823459+0000","last_change":"2024-09-15T02:48:29.823846+0000","last_active":"2024-09-15T02:48:29.823459+0000","last_peered":"2024-09-15T02:48:29.823459+0000","last_clean":"2024-09-15T02:48:29.823459+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:48:29.823459+0000","last_undegraded":"2024-09-15T02:48:29.823459+0000","last_fullsized":"2024-09-15T02:48:29.823459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:26.621594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418653+0000","last_change":"2024-09-15T02:48:29.418653+0000","last_active":"2024-09-15T02:48:29.418653+0000","last_peered":"2024-09-15T02:48:29.418653+0000","last_clean":"2024-09-15T02:48:29.418653+0000","last_became_active":"2024-09-15T02:48:27.398299+0000","last_became_peered":"2024-09-15T02:48:27.398299+0000","last_unstale":"2024-09-15T02:48:29.418653+0000","last_undegraded":"2024-09-15T02:48:29.418653+0000","last_fullsized":"2024-09-15T02:48:29.418653+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.751755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.420449+0000","last_change":"2024-09-15T02:48:29.420976+0000","last_active":"2024-09-15T02:48:29.420449+0000","last_peered":"2024-09-15T02:48:29.420449+0000","last_clean":"2024-09-15T02:48:29.420449+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:48:29.420449+0000","last_undegraded":"2024-09-15T02:48:29.420449+0000","last_fullsized":"2024-09-15T02:48:29.420449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:33.874284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.418630+0000","last_change":"2024-09-15T02:48:29.418940+0000","last_active":"2024-09-15T02:48:29.418630+0000","last_peered":"2024-09-15T02:48:29.418630+0000","last_clean":"2024-09-15T02:48:29.418630+0000","last_became_active":"2024-09-15T02:48:27.395014+0000","last_became_peered":"2024-09-15T02:48:27.395014+0000","last_unstale":"2024-09-15T02:48:29.418630+0000","last_undegraded":"2024-09-15T02:48:29.418630+0000","last_fullsized":"2024-09-15T02:48:29.418630+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:45.306020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011443880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T02:48:29.636075+0000","last_change":"2024-09-15T02:48:29.636494+0000","last_active":"2024-09-15T02:48:29.636075+0000","last_peered":"2024-09-15T02:48:29.636075+0000","last_clean":"2024-09-15T02:48:29.636075+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:48:29.636075+0000","last_undegraded":"2024-09-15T02:48:29.636075+0000","last_fullsized":"2024-09-15T02:48:29.636075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:30.837572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T02:48:28.637141+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:48:28.637141+0000","last_peered":"2024-09-15T02:48:28.637141+0000","last_clean":"2024-09-15T02:48:28.637141+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:48:28.637141+0000","last_undegraded":"2024-09-15T02:48:28.637141+0000","last_fullsized":"2024-09-15T02:48:28.637141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.005063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.415}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.278}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6989999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.577}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5139999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0779999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":127357,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7070000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.149}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9470000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.992}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]}]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,4,5,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0329999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1709999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9630000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297640,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074200,"statfs":{"total":96636764160,"available":96331980800,"internally_reserved":0,"allocated":286720,"data_stored":127954,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":640.51300000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.101}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127376,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.198}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.978}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":656863,"data_compressed":20370,"data_compressed_allocated":61440,"data_compressed_original":569344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7829999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297636,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":282624,"data_stored":127935,"data_compressed":12765,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:48:33.863 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T02:48:33.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-15T02:48:34.165 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 29 2024-09-15T02:48:34.165 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 24e12693-a4e4-4f9a-bb1d-999f41be75a2 2024-09-15T02:48:34.165 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-15T02:48:10.478850+0000 2024-09-15T02:48:34.165 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-09-15T02:48:29.405812+0000 2024-09-15T02:48:34.165 INFO:teuthology.orchestra.run.smithi006.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:crush_version 5 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:full_ratio 0.95 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:backfillfull_ratio 0.9 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:nearfull_ratio 0.85 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:require_min_compat_client luminous 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:min_compat_client jewel 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:require_osd_release quincy 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:stretch_mode_enabled false 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 29 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout: removed_snaps_queue [2~1] 2024-09-15T02:48:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:max_osd 16 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6816/22132,v1:172.21.15.6:6817/22132] [v2:172.21.15.6:6818/22132,v1:172.21.15.6:6819/22132] exists,up 2d28679c-afcb-472a-8a92-b30cbeb553a7 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/22077,v1:172.21.15.118:6810/22077] [v2:172.21.15.118:6812/22077,v1:172.21.15.118:6814/22077] exists,up a3111517-1d9f-4c6b-a838-8eb5e07f8c39 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6810/22074,v1:172.21.15.152:6813/22074] [v2:172.21.15.152:6816/22074,v1:172.21.15.152:6818/22074] exists,up 5a893db6-a9ac-4c69-ac36-57826eab4e3d 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6823/22466,v1:172.21.15.191:6827/22466] [v2:172.21.15.191:6828/22466,v1:172.21.15.191:6829/22466] exists,up ab0d2e8a-aedd-4d8b-a5d4-881317345d27 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6800/22131,v1:172.21.15.6:6801/22131] [v2:172.21.15.6:6802/22131,v1:172.21.15.6:6803/22131] exists,up 3edf8310-01a7-4377-a469-b376971149ea 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/22078,v1:172.21.15.118:6801/22078] [v2:172.21.15.118:6802/22078,v1:172.21.15.118:6803/22078] exists,up 8000b2e4-ca30-4591-88f2-9b0eee27cbca 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6800/22076,v1:172.21.15.152:6801/22076] [v2:172.21.15.152:6802/22076,v1:172.21.15.152:6803/22076] exists,up 7f8ecd2a-7829-496b-b349-5fdebbb6962a 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6802/22456,v1:172.21.15.191:6803/22456] [v2:172.21.15.191:6804/22456,v1:172.21.15.191:6805/22456] exists,up d94d0835-cb78-46f5-ae93-1b6d60a9ad84 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6808/22164,v1:172.21.15.6:6809/22164] [v2:172.21.15.6:6810/22164,v1:172.21.15.6:6811/22164] exists,up 61e692a5-d458-4bbc-8678-66f2c7b6bfe2 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6809/22079,v1:172.21.15.118:6811/22079] [v2:172.21.15.118:6813/22079,v1:172.21.15.118:6815/22079] exists,up b9a7b347-f334-4ccc-82ad-d6d67b366485 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6809/22075,v1:172.21.15.152:6812/22075] [v2:172.21.15.152:6815/22075,v1:172.21.15.152:6819/22075] exists,up f446f37d-a5c3-4e59-88f9-5a0bb818b35b 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6818/22467,v1:172.21.15.191:6819/22467] [v2:172.21.15.191:6820/22467,v1:172.21.15.191:6821/22467] exists,up 78916195-e38b-4ae2-a160-6feb89ec76ff 2024-09-15T02:48:34.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6824/22165,v1:172.21.15.6:6825/22165] [v2:172.21.15.6:6826/22165,v1:172.21.15.6:6827/22165] exists,up 778d9a09-0481-4af0-b838-ceeabc6c0af2 2024-09-15T02:48:34.168 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6824/22080,v1:172.21.15.118:6825/22080] [v2:172.21.15.118:6826/22080,v1:172.21.15.118:6827/22080] exists,up 5167c2e2-7879-4a97-9966-f0621d9ced72 2024-09-15T02:48:34.168 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] [v2:172.21.15.152:6814/22077,v1:172.21.15.152:6817/22077] exists,up 7b4d6903-aa33-4280-89f1-05c499290d8f 2024-09-15T02:48:34.168 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] [v2:172.21.15.191:6812/22457,v1:172.21.15.191:6813/22457] exists,up baf2647e-ce90-42a7-82ae-ec7578ef31da 2024-09-15T02:48:34.180 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6816/22132,v1:172.21.15.6:6817/22132] [v2:172.21.15.6:6818/22132,v1:172.21.15.6:6819/22132] exists,up 2d28679c-afcb-472a-8a92-b30cbeb553a7', 'osd.1 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/22077,v1:172.21.15.118:6810/22077] [v2:172.21.15.118:6812/22077,v1:172.21.15.118:6814/22077] exists,up a3111517-1d9f-4c6b-a838-8eb5e07f8c39', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6810/22074,v1:172.21.15.152:6813/22074] [v2:172.21.15.152:6816/22074,v1:172.21.15.152:6818/22074] exists,up 5a893db6-a9ac-4c69-ac36-57826eab4e3d', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6823/22466,v1:172.21.15.191:6827/22466] [v2:172.21.15.191:6828/22466,v1:172.21.15.191:6829/22466] exists,up ab0d2e8a-aedd-4d8b-a5d4-881317345d27', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6800/22131,v1:172.21.15.6:6801/22131] [v2:172.21.15.6:6802/22131,v1:172.21.15.6:6803/22131] exists,up 3edf8310-01a7-4377-a469-b376971149ea', 'osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/22078,v1:172.21.15.118:6801/22078] [v2:172.21.15.118:6802/22078,v1:172.21.15.118:6803/22078] exists,up 8000b2e4-ca30-4591-88f2-9b0eee27cbca', 'osd.6 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6800/22076,v1:172.21.15.152:6801/22076] [v2:172.21.15.152:6802/22076,v1:172.21.15.152:6803/22076] exists,up 7f8ecd2a-7829-496b-b349-5fdebbb6962a', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6802/22456,v1:172.21.15.191:6803/22456] [v2:172.21.15.191:6804/22456,v1:172.21.15.191:6805/22456] exists,up d94d0835-cb78-46f5-ae93-1b6d60a9ad84', 'osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6808/22164,v1:172.21.15.6:6809/22164] [v2:172.21.15.6:6810/22164,v1:172.21.15.6:6811/22164] exists,up 61e692a5-d458-4bbc-8678-66f2c7b6bfe2', 'osd.9 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6809/22079,v1:172.21.15.118:6811/22079] [v2:172.21.15.118:6813/22079,v1:172.21.15.118:6815/22079] exists,up b9a7b347-f334-4ccc-82ad-d6d67b366485', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6809/22075,v1:172.21.15.152:6812/22075] [v2:172.21.15.152:6815/22075,v1:172.21.15.152:6819/22075] exists,up f446f37d-a5c3-4e59-88f9-5a0bb818b35b', 'osd.11 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6818/22467,v1:172.21.15.191:6819/22467] [v2:172.21.15.191:6820/22467,v1:172.21.15.191:6821/22467] exists,up 78916195-e38b-4ae2-a160-6feb89ec76ff', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6824/22165,v1:172.21.15.6:6825/22165] [v2:172.21.15.6:6826/22165,v1:172.21.15.6:6827/22165] exists,up 778d9a09-0481-4af0-b838-ceeabc6c0af2', 'osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6824/22080,v1:172.21.15.118:6825/22080] [v2:172.21.15.118:6826/22080,v1:172.21.15.118:6827/22080] exists,up 5167c2e2-7879-4a97-9966-f0621d9ced72', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] [v2:172.21.15.152:6814/22077,v1:172.21.15.152:6817/22077] exists,up 7b4d6903-aa33-4280-89f1-05c499290d8f', 'osd.15 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] [v2:172.21.15.191:6812/22457,v1:172.21.15.191:6813/22457] exists,up baf2647e-ce90-42a7-82ae-ec7578ef31da'] 2024-09-15T02:48:34.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-15T02:48:34.282 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mon.a", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster": "ceph", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "admin_socket_mode": "", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_client_required": "cephx, none", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_cluster_required": "cephx", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_debug": "true", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_service_required": "cephx", 2024-09-15T02:48:34.283 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_supported": "cephx", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio": "true", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_poll_ms": "250", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_reap_max": "16", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_async_discard": "false", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_block_size": "4096", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio": "false", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_inflight_ios": "false", 2024-09-15T02:48:34.284 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_enable_discard": "false", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_flock_retry": "3", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_inject_crash": "0", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring": "false", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring_hipri": "false", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-15T02:48:34.285 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_type": "", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_alloc_size": "1048576", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_allocator": "hybrid", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_buffered_io": "true", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_for_zeros": "false", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_compact_log_sync": "false", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-15T02:48:34.286 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_max_log_runway": "4194304", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_max_prefetch": "1048576", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_min_flush_size": "524288", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_min_log_runway": "1048576", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_replay_recovery": "false", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_sync_write": "false", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_allocation_from_file": "true", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_allocator": "hybrid", 2024-09-15T02:48:34.287 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_async_db_compaction": "true", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_create": "true", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_create": "false", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_path": "", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_size": "0", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_path": "", 2024-09-15T02:48:34.288 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_preallocate_file": "false", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_size": "107374182400", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_create": "false", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_path": "", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_size": "100663296", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs": "true", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-15T02:48:34.289 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_autotune": "true", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size": "0", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_type": "2q", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-15T02:48:34.290 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_clone_cow": "true", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_mode": "none", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_csum_type": "crc32c", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_freelist": "false", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-15T02:48:34.291 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-15T02:48:34.292 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_small_allocations": "0", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_default_buffered_read": "true", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_default_buffered_write": "false", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-15T02:48:34.293 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fail_eio": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mount": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_umount": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-15T02:48:34.294 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_ignore_data_csum": "false", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_op_age": "5.000000", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_alloc_size": "0", 2024-09-15T02:48:34.295 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size": "0", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_deferred_txc": "32", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size": "0", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_nid_prealloc": "1024", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-15T02:48:34.296 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_retry_disk_reads": "3", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_cf": "true", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_mem": "512", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-15T02:48:34.297 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_tracing": "false", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-15T02:48:34.298 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_zero_block_detection": "false", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-15T02:48:34.299 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_cluster_require_signatures": "false", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_cluster_require_version": "2", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_require_signatures": "false", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_require_version": "2", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_service_require_signatures": "false", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_service_require_version": "2", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_sign_messages": "true", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "chdir": "", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "client_acl_type": "", 2024-09-15T02:48:34.300 INFO:teuthology.orchestra.run.smithi006.stdout: "client_asio_thread_count": "2", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_cache_mid": "0.750000", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_cache_size": "16384", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_caps_release_delay": "5", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_check_pool_perm": "true", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_force_sync_read": "false", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_getattr_caps": "false", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_inject_features": "", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_inject_tick_delay": "0", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_die_on_failed_remount": "false", 2024-09-15T02:48:34.301 INFO:teuthology.orchestra.run.smithi006.stdout: "client_dirsize_rbytes": "true", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_force_lazyio": "false", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_fs": "", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_inject_release_failure": "false", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_max_inline_size": "4096", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mds_namespace": "", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_metadata": "", 2024-09-15T02:48:34.302 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_gid": "-1", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_timeout": "300", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_uid": "-1", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mountpoint": "/", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_notify_timeout": "10", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc": "true", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_dirty": "104857600", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_objects": "1000", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_size": "209715200", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_target_dirty": "8388608", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_permissions": "true", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_quota": "true", 2024-09-15T02:48:34.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_quota_df": "true", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_max_bytes": "0", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_max_periods": "4", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_min": "131072", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_reconnect_stale": "false", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_shutdown_timeout": "30", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_snapdir": ".snap", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_tick_interval": "1", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_trace": "", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_try_dentry_invalidate": "false", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_use_faked_inos": "false", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_use_random_mds": "false", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog": "false", 2024-09-15T02:48:34.304 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog_port": "12201", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_monitors": "default=true", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog": "false", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog_level": "info", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "-", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_network": "", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_network_interface": "", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_isal": "false", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_level": "5", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_winsize": "-15", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zstd_level": "1", 2024-09-15T02:48:34.305 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_alien_op_num_threads": "6", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location_hook": "", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location_hook_timeout": "10", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "daemonize": "false", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_alienstore": "0/5", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_allow_any_pool_priority": "false", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asok": "1/5", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asok_assert_abort": "false", 2024-09-15T02:48:34.306 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asserts_on_shutdown": "true", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_auth": "20/20", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bdev": "1/3", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bluefs": "1/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bluestore": "1/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_buffer": "0/1", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ceph_exporter": "1/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cephfs_mirror": "0/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cephsqlite": "0/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_civetweb": "1/10", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_client": "0/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_compressor": "1/5", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_context": "0/1", 2024-09-15T02:48:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crush": "1/1", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crypto": "1/5", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_deliberately_leak_memory": "false", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_disable_randomized_ping": "false", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_dpdk": "1/5", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_eventtrace": "1/5", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_filer": "0/1", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_filestore": "1/3", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_finisher": "1/1", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_fuse": "1/5", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_heartbeat_testing_span": "0", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_heartbeatmap": "1/5", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-15T02:48:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_javaclient": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_journal": "1/3", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_journaler": "0/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_kstore": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_leveldb": "4/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_lockdep": "0/1", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mclock": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_balancer": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_locker": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_log": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_log_expire": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_migrator": "1/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_memdb": "4/5", 2024-09-15T02:48:34.309 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mgr": "2/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mgrc": "1/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mon": "20/20", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_monc": "0/10", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ms": "1/1", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_none": "0/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objclass": "0/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objectcacher": "0/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objecter": "0/1", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_optracker": "0/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_osd": "1/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_paxos": "20/20", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_perfcounter": "1/5", 2024-09-15T02:48:34.310 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_prioritycache": "1/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rados": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_mirror": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_pwl": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_replay": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_refs": "0/0", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_reserver": "1/1", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw": "1/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_datacache": "1/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_sync": "1/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rocksdb": "4/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_cache": "0/5", 2024-09-15T02:48:34.311 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_cleaner": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_device": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_journal": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_lba": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_odata": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_omap": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_onode": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_tm": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_striper": "0/1", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_test": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_throttle": "1/1", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_timer": "0/1", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_tp": "0/5", 2024-09-15T02:48:34.312 INFO:teuthology.orchestra.run.smithi006.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "device_failure_prediction_mode": "none", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_graylog": "false", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_journald": "false", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_stderr": "true", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_syslog": "false", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event_tracing": "false", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_addr": "0.0.0.0", 2024-09-15T02:48:34.313 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_http_port": "9926", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_prio_limit": "5", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_sort_metrics": "true", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_stats_period": "5", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "fake_statfs_for_testing": "0", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "fatal_signal_handlers": "true", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filer_max_purge_ops": "10", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filer_max_truncate_ops": "128", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_apply_finisher_threads": "1", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_blackhole": "false", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_btrfs_clone_range": "true", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_btrfs_snap": "true", 2024-09-15T02:48:34.314 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_caller_concurrency": "10", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_collect_device_partition_information": "true", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_commit_timeout": "600.000000", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_inject_read_err": "false", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_omap_check": "false", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_verify_split": "false", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_dump_file": "", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fadvise": "true", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fail_eio": "true", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fd_cache_shards": "16", 2024-09-15T02:48:34.315 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fd_cache_size": "128", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fiemap": "false", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fiemap_threshold": "4096", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_inject_stall": "0", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_parallel": "false", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_trailing": "false", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_writeahead": "false", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_kill_at": "0", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-15T02:48:34.316 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs": "0", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size": "0", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_merge_threshold": "-10", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-15T02:48:34.317 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_odsync_write": "false", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_backend": "rocksdb", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_backend_path": "", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_thread_timeout": "60", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_threads": "2", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_punch_hole": "false", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-15T02:48:34.318 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_ops": "50", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_seek_data_hole": "false", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_sloppy_crc": "false", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_splice": "false", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_split_multiple": "2", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_split_rand_factor": "20", 2024-09-15T02:48:34.319 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_update_to": "1000", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_enable": "true", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-15T02:48:34.320 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_xfs_extsize": "false", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_zfs_snap": "false", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fio_dir": "/tmp/fio", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "24e12693-a4e4-4f9a-bb1d-999f41be75a2", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_allow_other": "true", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_atomic_o_trunc": "true", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_big_writes": "true", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_debug": "false", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_default_permissions": "false", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_disable_pagecache": "false", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_max_write": "0", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_multithreaded": "true", 2024-09-15T02:48:34.321 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_require_active_mds": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_set_user_groups": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_move": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_read": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_write": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_use_invalidate_cb": "true", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "gss_target_name": "ceph", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_file": "", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_inject_failure": "0", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_interval": "5", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "host": "smithi006", 2024-09-15T02:48:34.322 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-15T02:48:34.323 INFO:teuthology.orchestra.run.smithi006.stdout: "inject_early_sigterm": "false", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "jaeger_tracing_enable": "false", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_aio": "true", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_align_min_size": "65536", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_block_align": "true", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_block_size": "4096", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_dio": "true", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_discard": "false", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_force_aio": "false", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_ignore_corruption": "false", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_max_write_bytes": "10485760", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_max_write_entries": "100", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_replay_from": "0", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-15T02:48:34.324 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_write_header_frequency": "0", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_zero_on_create": "false", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_prefetch_periods": "10", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_prezero_periods": "5", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_write_head_interval": "15", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "keyfile": "", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_backend": "rocksdb", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_default_stripe_size": "65536", 2024-09-15T02:48:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_fsck_on_mount": "false", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_max_bytes": "67108864", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_max_ops": "512", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_nid_prealloc": "1024", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_onode_map_size": "1024", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_sync_submit_transaction": "false", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_sync_transaction": "false", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_block_size": "65536", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_bloom_size": "0", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_cache_size": "536870912", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_compact_on_mount": "false", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_compression": "false", 2024-09-15T02:48:34.326 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_log": "", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_max_open_files": "0", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_paranoid": "false", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "librados_thread_count": "2", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "lockdep": "false", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "lockdep_force_backtrace": "false", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_coarse_timestamps": "true", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_flush_on_exit": "false", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_graylog_host": "127.0.0.1", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_graylog_port": "12201", 2024-09-15T02:48:34.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_max_new": "1000", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_max_recent": "10000", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_stderr_prefix": "", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_stop_at_utilization": "0.970000", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": "true", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_graylog": "false", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_journald": "false", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_stderr": "false", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_syslog": "false", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "max_rotating_auth_attempts": "10", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_action_on_write_error": "1", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_alternate_name_max": "8192", 2024-09-15T02:48:34.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_asio_thread_count": "2", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_export_pin": "true", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_dirs": "true", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_interval": "5", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_interval": "10", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_max": "-1", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_max_until": "-1", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_merge_size": "50", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_midchunk": "0.300000", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-15T02:48:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_min_start": "0.200000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_minchunk": "0.001000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_mode": "0", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_need_max": "1.200000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_need_min": "0.800000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_bits": "3", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_size": "10000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_target_decay": "10.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_grace": "15.000000", 2024-09-15T02:48:34.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_interval": "4.000000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_mid": "0.700000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_release_free_interval": "10", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_reservation": "0.050000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_interval": "1", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_threshold": "262144", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_prealloc_inos": "1000", 2024-09-15T02:48:34.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_connect_bootstrapping": "false", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_damage_table_max_entries": "10000", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_auth_pins": "false", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_frag": "false", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_scatterstat": "false", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_subtrees": "false", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_decay_halflife": "5.000000", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_default_dir_hash": "2", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_defer_session_stale": "true", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_deny_all_reconnect": "false", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_keys_per_op": "16384", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_max_commit_size": "10", 2024-09-15T02:48:34.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_max_entries": "0", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_on_map": "false", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_early_reply": "true", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_enable_op_tracker": "true", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_enforce_unique_name": "true", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_random": "true", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-15T02:48:34.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_health_summarize_threshold": "10", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_health_dummy": "false", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_migrator_session_race": "false", 2024-09-15T02:48:34.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-15T02:48:34.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-15T02:48:34.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-15T02:48:34.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_join_fs": "", 2024-09-15T02:48:34.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_journal_format": "1", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_create_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_export_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_import_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_expire_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_replay_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_link_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_mdstable_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_openc_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_rename_at": "0", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_events_per_segment": "1024", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_max_events": "-1", 2024-09-15T02:48:34.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_max_segments": "128", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_pause": "false", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_segment_size": "0", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_warn_factor": "2.000000", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_caps_per_client": "1048576", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_completed_flushes": "100000", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_completed_requests": "100000", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_export_size": "20971520", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_file_recover": "32", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_files": "64", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_ops": "8192", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-15T02:48:34.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_snaps_per_dir": "100", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_metrics_update_interval": "2", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_min_caps_per_client": "100", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_min_caps_working_set": "10000", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_numa_node": "-1", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_complaint_time": "30.000000", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_duration": "600", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_size": "20", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_slow_op_size": "20", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-15T02:48:34.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_log_threshold": "5", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_ping_grace": "15", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_ping_interval": "5", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_caps": "30000", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_warning_threshold": "262144", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_replay_interval": "1.000000", 2024-09-15T02:48:34.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_root_ino_gid": "0", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_root_ino_uid": "0", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-15T02:48:34.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_shutdown_check": "0", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_skip_ino": "0", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_max_uid": "4294967294", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_min_uid": "0", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_rstat": "false", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_standby_replay_damaged": "false", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_symlink_recovery": "true", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_thrash_exports": "0", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_thrash_fragments": "0", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_tick_interval": "5.000000", 2024-09-15T02:48:34.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_valgrind_exit": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_verify_backtrace": "1", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_verify_scatter": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_wipe_sessions": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool_debug": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_device_bytes": "1073741824", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_page_set": "false", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_page_size": "65536", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_bytes": "134217728", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_messages": "512", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-15T02:48:34.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_max_pg_num_change": "128", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mds_bytes": "134217728", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mds_messages": "128", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mon_bytes": "134217728", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mon_messages": "128", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_osd_bytes": "536870912", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_osd_messages": "8192", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_pool": "true", 2024-09-15T02:48:34.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_standby_modules": "true", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_stats_period": "5", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_stats_threshold": "5", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_tick_period": "2", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_allow_pool_delete": "true", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_allow_pool_size_one": "true", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_auth_validate_all_caps": "true", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-15T02:48:34.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_bytes": "104857600", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_directed_command_retry": "5", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_parallel": "3", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_log_interval": "1.000000", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-15T02:48:34.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_ping_interval": "10.000000", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_level": "debug", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_file": "true", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_journald": "false", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-15T02:48:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_bootstrap": "false", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_start": "false", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_trim": "true", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cpu_threads": "4", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_crush_min_required_version": "hammer", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_daemon_bytes": "419430400", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_avail_crit": "5", 2024-09-15T02:48:34.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_avail_warn": "5", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_size_warn": "16106127360", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_json": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_transactions": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_extra_checks": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_pacific": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_quincy": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-15T02:48:34.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_down_mkfs_grace": "120", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_election_default_strategy": "3", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_election_timeout": "5.000000", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ping_divisor": "2", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_enable_op_tracker": "true", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_fake_pool_delete": "false", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_force_quorum_join": "false", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_globalid_prealloc": "10000", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_detail_to_clog": "false", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_log_update_period": "5", 2024-09-15T02:48:34.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_max_detail": "50", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog": "true", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog_interval": "600", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_host": "172.21.15.6,172.21.15.118,172.21.15.152", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_host_override": "", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_initial_members": "", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-15T02:48:34.351 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease": "5.000000", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_full_interval": "50", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_max": "10000", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_max_summary": "50", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_log_epochs": "500", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_osd": "10000", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_pg_per_osd": "10000", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_pool_pg_num": "65536", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-15T02:48:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_force_trim_to": "0", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_skip_sanity": "false", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_memory_autotune": "true", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_memory_target": "2147483648", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_beacon_grace": "30", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_digest_period": "5", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_inactive_grace": "60", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_min_osdmap_epochs": "500", 2024-09-15T02:48:34.353 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_complaint_time": "30", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_duration": "600", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_size": "20", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_slow_op_size": "20", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_log_threshold": "5", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_in": "false", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-15T02:48:34.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_cache_size": "500", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_down_out_interval": "600", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_force_trim_to": "0", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-15T02:48:34.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_down_reporters": "2", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_report_timeout": "900", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-15T02:48:34.356 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_stuck_threshold": "60", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-15T02:48:34.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_probe_timeout": "2.000000", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_max_change": "0.050000", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_max_osds": "4", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_interval": "300", 2024-09-15T02:48:34.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_max_keys": "100", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_timeout": "300", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_session_timeout": "300", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_smart_report_timeout": "5", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stat_smooth_intervals": "6", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_pool_min_size": "2", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_pool_size": "4", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_debug": "false", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-15T02:48:34.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_provider_kill_at": "0", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_requester_kill_at": "0", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_timeout": "60.000000", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_target_pg_per_osd": "100", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_tick_interval": "5", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_timecheck_interval": "300.000000", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_older_version_delay": "604800", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-15T02:48:34.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_misplaced": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_older_version": "true", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-15T02:48:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": "", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_op_threads": "3", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_cm": "false", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_device_name": "", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_dscp": "96", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_local_gid": "", 2024-09-15T02:48:34.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_port_num": "1", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_sl": "3", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_support_srq": "true", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_type": "ib", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_reap_threshold": "5", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_before_connect": "false", 2024-09-15T02:48:34.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_ipv4": "true", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_ipv6": "false", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_msgr1": "true", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_msgr2": "true", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_port_max": "7568", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_port_min": "6800", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_retry_count": "3", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_retry_delay": "5", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_client": "false", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mds": "false", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mgr": "false", 2024-09-15T02:48:34.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mon": "false", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_osd": "false", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_client_mode": "crc secure", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_cluster_mode": "crc secure", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_cluster_type": "", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_compress_secure": "false", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_connection_idle_timeout": "900", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_connection_ready_timeout": "10", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_crc_data": "true", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_crc_header": "true", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_bad_msg": "false", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_bug": "true", 2024-09-15T02:48:34.365 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_old_message": "true", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_skipped_message": "false", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_coremask": "0xF", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_enable_tso": "true", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hugepages": "", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-15T02:48:34.366 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_lro": "true", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_memory_channel": "4", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_pmd": "", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_port_id": "0", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dump_on_send": "false", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_initial_backoff": "0.200000", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_max": "1.000000", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_type": "osd", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_socket_failures": "2500", 2024-09-15T02:48:34.367 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_learn_addr_from_peer": "true", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_max_accept_failures": "4", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_max_backoff": "15.000000", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_client_mode": "secure crc", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_service_mode": "secure crc", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compress_min_size": "1024", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compress_mode": "none", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_pq_min_cost": "65536", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_public_type": "", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_service_mode": "crc secure", 2024-09-15T02:48:34.368 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_listen_backlog": "512", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_nodelay": "true", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_rcvbuf": "0", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_type": "async+posix", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "no_config_file": "false", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_completion_locks_per_session": "32", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inflight_ops": "1024", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-15T02:48:34.369 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_tick_interval": "5.000000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_timeout": "10.000000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "objectstore_blackhole": "false", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "openssl_engine_opts": "", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_delay_time": "5.000000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_hist_halflife": "1000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_max_low_ops": "2", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_max_ops": "4", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_slop": "0.020000", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-15T02:48:34.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_asio_thread_count": "2", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_async_recovery_min_cost": "100", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_scan_max": "512", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_scan_min": "64", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_degraded": "false", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_peering": "false", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_unfound": "true", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_beacon_report_interval": "300", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_duration": "30", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-15T02:48:34.371 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_max_block_size": "67108864", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_blkin_trace_all": "false", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_check_for_log_corruption": "false", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_update_on_start": "true", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_message_cap": "256", 2024-09-15T02:48:34.372 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_message_size_cap": "524288000", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_op_priority": "63", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_watch_timeout": "30", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_command_max_records": "256", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_compact_on_start": "false", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_update_on_start": "true", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_update_weight_set": "true", 2024-09-15T02:48:34.373 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_feed_pullee": "-1", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_misdirected_ops": "false", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_no_acting_change": "false", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_no_purge_strays": "false", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_op_order": "false", 2024-09-15T02:48:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_shutdown": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_snaps": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_keys": "1024", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-15T02:48:34.375 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_stride": "524288", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_default_notify_timeout": "30", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep": "0.000000", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_discard_disconnected_ops": "true", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_enable_op_tracker": "true", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-15T02:48:34.376 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_info": "true", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown": "true", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_function_tracing": "false", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_grace": "20", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_interval": "6", 2024-09-15T02:48:34.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_peers": "10", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_size": "2000", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_stale": "600", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_max_size": "100000", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_min_size": "1000", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-15T02:48:34.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal_size": "5120", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_kill_backfill_at": "0", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_cache_size": "50", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_dedup": "true", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_message_max": "40", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_share_max_epochs": "40", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_attr_name_len": "100", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_attr_size": "0", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_backfills": "1", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_markdown_count": "5", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_markdown_period": "600", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_name_len": "2048", 2024-09-15T02:48:34.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_namespace_len": "256", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_size": "134217728", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_blocked_by": "16", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_log_entries": "10000", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pgls": "1024", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_push_cost": "8388608", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_push_objects": "10", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_scrubs": "3", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-15T02:48:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_trimming_pgs": "2", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_write_op_reply_len": "64", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_write_size": "90", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_profile": "balanced", 2024-09-15T02:48:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_base": "805306368", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_cache_min": "134217728", 2024-09-15T02:48:34.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target": "4294967296", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target_autotune": "false", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_min_pg_log_entries": "250", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_min_recovery_priority": "0", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_report_interval": "5", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_num_cache_shards": "32", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_num_op_tracker_shard": "32", 2024-09-15T02:48:34.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_auto_affinity": "true", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_node": "-1", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_prefer_iface": "true", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objecter_finishers": "1", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore": "bluestore", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_fuse": "false", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_tracing": "false", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_complaint_time": "30.000000", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_duration": "600", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_size": "20", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_slow_op_size": "20", 2024-09-15T02:48:34.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_log_threshold": "5", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards": "0", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards_hdd": "5", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards_ssd": "8", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_pq_min_cost": "65536", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_queue_cut_off": "high", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-15T02:48:34.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_thread_timeout": "15", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_open_classes_on_start": "true", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_os_flags": "0", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_peering_op_priority": "255", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_delete_cost": "1048576", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_delete_priority": "5", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_trim_max": "10000", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_trim_min": "100", 2024-09-15T02:48:34.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-15T02:48:34.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flags": "0", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_min_size": "0", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pg_num": "32", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pgp_num": "0", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_size": "2", 2024-09-15T02:48:34.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_type": "replicated", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_push_per_object_cost": "1000", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recover_clone_overlap": "true", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_cost": "20971520", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active": "0", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-15T02:48:34.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_single_start": "1", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_op_priority": "3", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_priority": "5", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep": "0.000000", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_repair_during_recovery": "false", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_requested_scrub_priority": "120", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_auto_repair": "false", 2024-09-15T02:48:34.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_begin_hour": "0", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_begin_week_day": "0", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_chunk_max": "25", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_chunk_min": "5", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_cost": "52428800", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_during_recovery": "false", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_end_hour": "0", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_end_week_day": "0", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_event_cost": "4096", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_invalid_stats": "true", 2024-09-15T02:48:34.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_max_preemptions": "5", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_priority": "5", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_sleep": "0.000000", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_skip_data_digest": "false", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_smart_report_timeout": "5", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_cost": "1048576", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_priority": "5", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-15T02:48:34.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_target_transaction_size": "30", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-15T02:48:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tracing": "false", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_use_stale_snap": "false", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osdc_blkin_trace_all": "false", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_kill_at": "0", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_max_join_drift": "10", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_min": "500", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_min_wait": "0.050000", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_propose_interval": "1.000000", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_max": "500", 2024-09-15T02:48:34.394 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_min": "250", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_stash_full_interval": "25", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_trim_max": "500", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_trim_min": "250", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "perf": "true", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "-", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrv": "", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "public_bind_addr": "-", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "public_network": "", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "public_network_interface": "", 2024-09-15T02:48:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: "qat_compressor_enabled": "false", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_mon_op_timeout": "0", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_osd_op_timeout": "0", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_tracing": "false", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_atime_update_interval": "60", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_balance_parent_reads": "false", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_balance_snap_reads": "false", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blkin_trace_all": "false", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache": "true", 2024-09-15T02:48:34.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_policy": "writearound", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_size": "33554432", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_clone_copy_on_read": "false", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_compression_hint": "none", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_concurrent_management_ops": "10", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_clone_format": "auto", 2024-09-15T02:48:34.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_data_pool": "", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_features": "61", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_format": "2", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_map_options": "", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_order": "22", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_pool": "rbd", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_stripe_count": "0", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_stripe_unit": "0", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_enable_alloc_hint": "true", 2024-09-15T02:48:34.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_io_scheduler": "simple", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-15T02:48:34.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_order": "24", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_pool": "", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_splay_width": "4", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_localize_parent_reads": "false", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_localize_snap_reads": "false", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-15T02:48:34.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-15T02:48:34.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mtime_update_interval": "60", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_non_blocking_aio": "true", 2024-09-15T02:48:34.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_op_thread_timeout": "60", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_op_threads": "1", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_parent_cache_enabled": "false", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_plugins": "", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_burst": "0", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_limit": "0", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_exclude_ops": "0", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_burst": "0", 2024-09-15T02:48:34.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_limit": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-15T02:48:34.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_read_from_replica_policy": "default", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_skip_partial_discard": "true", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_tracing": "false", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_validate_names": "true", 2024-09-15T02:48:34.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_validate_pool": "true", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "restapi_base_url": "", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "restapi_log_level": "", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_acl_grants_max_num": "100", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_admin_entry": "admin", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_backend_store": "rados", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_barbican_url": "", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_beast_enable_async": "true", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-15T02:48:34.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_enabled": "true", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_expiry_interval": "900", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_lru_size": "10000", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_content_length_compat": "false", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_copy_obj_progress": "true", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cors_rules_max_num": "100", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cross_domain_policy": "", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_addr": "", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-15T02:48:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_password": "", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_username": "", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_require_ssl": "true", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-15T02:48:34.408 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_addr": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_auth": "token", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_namespace": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_prefix": "", 2024-09-15T02:48:34.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_token_file": "", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_buffersize": "524288", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_low_speed_time": "300", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-15T02:48:34.410 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_changes_size": "1000", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_num_shards": "128", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_window": "30", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-15T02:48:34.411 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-15T02:48:34.412 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dns_name": "", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dns_s3website_name": "", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dynamic_resharding": "true", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_gc_threads": "true", 2024-09-15T02:48:34.413 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_lc_threads": "true", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_ops_log": "false", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_quota_threads": "true", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_static_website": "false", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_usage_log": "false", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enforce_swift_acls": "true", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_exit_timeout_secs": "120", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_expose_bucket": "false", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_extended_http_attrs": "", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_frontends": "beast port=7480", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_deferred": "50", 2024-09-15T02:48:34.414 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_objs": "32", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_processor_period": "3600", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_init_timeout": "300", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-15T02:48:34.415 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_connection_idle": "300", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_domain": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_password": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_password_path": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_project": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_tenant": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_token": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_token_path": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_user": "", 2024-09-15T02:48:34.416 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_api_version": "2", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_domain": "", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_password": "", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_project": "", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_user": "", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_url": "", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_debug_interval": "-1", 2024-09-15T02:48:34.417 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_lock_max_time": "90", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_objs": "32", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_rules": "1000", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_worker": "3", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_thread_delay": "0", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_dnattr": "uid", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_searchfilter": "", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_uri": "", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-15T02:48:34.418 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_http_headers": "", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_object_name_utc": "false", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attr_name_len": "0", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attr_size": "0", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_chunk_size": "4194304", 2024-09-15T02:48:34.419 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_listing_results": "1000", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_notify_retries": "10", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_put_param_size": "1048576", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_put_size": "5368709120", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_slo_entries": "1000", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_md_log_max_shards": "64", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_mp_lock_max_time": "600", 2024-09-15T02:48:34.420 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_max_gc": "300", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-15T02:48:34.421 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_num_async_rados_threads": "32", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_num_control_oids": "8", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_numa_node": "-1", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_chunk_size": "100", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_gc_interval": "600", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_thread_timeout": "600", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_tracing": "false", 2024-09-15T02:48:34.422 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_token": "", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_url": "", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_verify_ssl": "true", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_file_path": "", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_rados": "true", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_socket_path": "", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_push_interval": "2.000000", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_print_continue": "true", 2024-09-15T02:48:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_tracing": "false", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm": "", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm_id": "", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_region": "", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-15T02:48:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_request_uri": "", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_batch_size": "64", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_max_aio": "128", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_num_logs": "16", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_thread_interval": "600", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_resolve_cname": "false", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_run_sync_thread": "true", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-15T02:48:34.425 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_scheduler_type": "throttler", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_script_uri": "", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_service_provider_name": "", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_client_id": "", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_client_secret": "", 2024-09-15T02:48:34.426 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_entry": "sts", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_key": "sts", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_min_session_duration": "900", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_token_introspection_url": "", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_account_in_url": "false", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_auth_entry": "auth", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_auth_url": "", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_custom_header": "", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_need_stats": "true", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_tenant_name": "", 2024-09-15T02:48:34.427 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_token_expiration": "86400", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_url": "", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_url_prefix": "swift", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_lease_period": "120", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-15T02:48:34.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_thread_pool_size": "512", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_comment": "", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_createby": "", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_encoding": "", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_flag": "false", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_origin": "", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_tracker": "", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_trust_forwarded_https": "false", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_max_shards": "32", 2024-09-15T02:48:34.429 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_max_user_shards": "1", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_use_opa_authz": "false", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_max_buckets": "1000", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_unique_email": "true", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_verify_ssl": "true", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone": "", 2024-09-15T02:48:34.430 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone_id": "", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup": "", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup_id": "", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_block_size": "4096", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_size": "536870912", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-15T02:48:34.431 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_index_type": "binary_search", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_partition_filters": "false", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_perf": "false", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-15T02:48:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "run_dir": "/var/run/ceph", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_block_create": "true", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_cache_lru_size": "67108864", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_default_max_object_size": "16777216", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_device_size": "107374182400", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_capacity": "16", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_segment_size": "67108864", 2024-09-15T02:48:34.433 INFO:teuthology.orchestra.run.smithi006.stdout: "setgroup": "", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "setuser": "", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "setuser_match_path": "", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "thp": "false", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "threadpool_default_timeout": "60", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout: "throttler_perf_counter": "true" 2024-09-15T02:48:34.434 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:34.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-15T02:48:34.587 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a: {} 2024-09-15T02:48:34.610 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b: {} 2024-09-15T02:48:34.634 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c: {} 2024-09-15T02:48:34.677 INFO:teuthology.orchestra.run.smithi006.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T02:48:34.678 INFO:teuthology.orchestra.run.smithi006.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T02:48:34.678 INFO:teuthology.orchestra.run.smithi006.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T02:48:34.678 DEBUG:teuthology.orchestra.run.smithi152:> type ceph-objectstore-tool 2024-09-15T02:48:34.684 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T02:48:34.684 DEBUG:teuthology.orchestra.run.smithi118:> type ceph-objectstore-tool 2024-09-15T02:48:34.690 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T02:48:34.690 DEBUG:teuthology.orchestra.run.smithi006:> type ceph-objectstore-tool 2024-09-15T02:48:34.696 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T02:48:34.697 DEBUG:teuthology.orchestra.run.smithi191:> type ceph-objectstore-tool 2024-09-15T02:48:34.703 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T02:48:34.703 INFO:teuthology.run_tasks:Running task rados... 2024-09-15T02:48:34.719 INFO:tasks.rados:Beginning rados... 2024-09-15T02:48:34.719 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-15T02:48:34.719 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:48:34.719 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-15T02:48:34.720 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-15T02:48:34.727 INFO:tasks.rados:joining rados 2024-09-15T02:48:34.727 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-15T02:48:34.727 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T02:48:34.727 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:48:34.728 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-15T02:48:34.728 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T02:48:34.728 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-15T02:48:34.728 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:48:34.728 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-15T02:48:34.729 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-15T02:48:34.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:48:34.729 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-15T02:48:34.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:48:34.730 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-15T02:48:34.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:48:34.730 INFO:tasks.rados:clients are ['client.0'] 2024-09-15T02:48:34.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-15T02:48:34.746 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:34.741+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:34.791 INFO:tasks.ceph.osd.10:Stopped 2024-09-15T02:48:34.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 10 2024-09-15T02:48:34.831 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:34.826+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:34.849 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:48:34.849 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:34.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:48:34.910 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:34.932 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:34.930+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:34.945 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:34.964 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:34.965 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:34.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:48:34.976 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:35.003 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:35.033 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:35.029+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:35.038 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:35.065 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:35.066 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:35.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:48:35.135 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:35.127+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:35.143 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-15T02:48:35.135+0000 7f08e39de700 -1 mon.a@0(leader).osd e30 definitely_dead 0 2024-09-15T02:48:35.168 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:35.168 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:35.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:48:35.236 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:35.229+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:35.316 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:35.316 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:35.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:48:35.337 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:35.329+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:35.426 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:35.426 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:35.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:48:35.438 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:35.433+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:35.479 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:48:35.480 INFO:teuthology.orchestra.run.smithi006.stderr:marked down osd.10. 2024-09-15T02:48:35.480 INFO:teuthology.orchestra.run.smithi006.stderr:load: shec 2024-09-15T02:48:35.497 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-15T02:48:35.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 10 2024-09-15T02:48:35.498 INFO:tasks.rados:starting run 0 out of 1 2024-09-15T02:48:35.498 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-15T02:48:35.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-15T02:48:35.538 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:35.533+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:35.541 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:35.541 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:35.541 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:48:35.541 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:35.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:48:35.639 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:35.640 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:35.640 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:48:35.640 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:48:35.640 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:35.640 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:35.633+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:35.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:48:35.741 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:35.738+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4311.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:28.361067+0000", 2024-09-15T02:48:35.766 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.402455389, 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00045800000000000002, 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4311", 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 2 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361067+0000", 2024-09-15T02:48:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361067+0000", 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361082+0000", 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4494999999999999e-05 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361098+0000", 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5583999999999999e-05 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361127+0000", 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9767000000000001e-05 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361144+0000", 2024-09-15T02:48:35.771 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6829000000000001e-05 2024-09-15T02:48:35.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361264+0000", 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012000100000000001 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361525+0000", 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026132400000000001 2024-09-15T02:48:35.773 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4311.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:28.361924+0000", 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.4015992810000002, 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00023242799999999999, 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4311", 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 3 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361924+0000", 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:35.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361924+0000", 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361926+0000", 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1090000000000001e-06 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361938+0000", 2024-09-15T02:48:35.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2480000000000001e-05 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361971+0000", 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2432999999999999e-05 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.361983+0000", 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2478e-05 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.777 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.362040+0000", 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6876000000000002e-05 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.362156+0000", 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011605200000000001 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4311.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:28.404544+0000", 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.3589786229999996, 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033046289999999999, 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4311", 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 5 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404544+0000", 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404544+0000", 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404547+0000", 2024-09-15T02:48:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.903e-06 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404564+0000", 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6517000000000001e-05 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404602+0000", 2024-09-15T02:48:35.781 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8355999999999997e-05 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404615+0000", 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2916e-05 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404704+0000", 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9290999999999996e-05 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:35.782 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404920+0000", 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000215718 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 9", 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.405364+0000", 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044384099999999999 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.406736+0000", 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013719660000000001 2024-09-15T02:48:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407694+0000", 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00095846600000000005 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407793+0000", 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8686999999999999e-05 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407849+0000", 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5967999999999997e-05 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.4311.0:5 2.2 e27/26 ondisk, result = 0)", 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:28.407449+0000", 2024-09-15T02:48:35.785 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.3560742809999997, 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00041356999999999999, 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407449+0000", 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407449+0000", 2024-09-15T02:48:35.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407450+0000", 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6959999999999999e-06 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407460+0000", 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4539999999999996e-06 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407536+0000", 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6783e-05 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407548+0000", 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1211e-05 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407663+0000", 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011547799999999999 2024-09-15T02:48:35.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407674+0000", 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0532e-05 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407862+0000", 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018841600000000001 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.789 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4311.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:29.419678+0000", 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.3438450189999998, 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0027426759999999999, 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4311", 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 7 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419678+0000", 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419678+0000", 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.791 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419681+0000", 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9119999999999998e-06 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419695+0000", 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4532000000000001e-05 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419719+0000", 2024-09-15T02:48:35.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3504000000000001e-05 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419747+0000", 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8011999999999999e-05 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419811+0000", 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3761000000000002e-05 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419908+0000", 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7108999999999995e-05 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 9", 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.420081+0000", 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017289900000000001 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.421213+0000", 2024-09-15T02:48:35.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001132498 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422332+0000", 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001118513 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422383+0000", 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0998000000000002e-05 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422420+0000", 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7938000000000002e-05 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.4311.0:7 2.2 e28/26 ondisk, result = 0)", 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:29.422169+0000", 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.3413537949999998, 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025723400000000001, 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:35.796 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422169+0000", 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422169+0000", 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422171+0000", 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5e-06 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422180+0000", 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.037e-06 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422250+0000", 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0755999999999998e-05 2024-09-15T02:48:35.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422261+0000", 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0231000000000001e-05 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422315+0000", 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4882000000000001e-05 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422323+0000", 2024-09-15T02:48:35.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5939999999999996e-06 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422426+0000", 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010323399999999999 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:35.800 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:35.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:48:35.845 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:35.839+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:35.878 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-15T02:48:35.871+0000 7f08e39de700 -1 mon.a@0(leader).osd e31 definitely_dead 0 2024-09-15T02:48:35.878 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:35.905 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:35.910 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:48:35.910 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:35.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:48:35.937 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:35.943 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:35.941+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:35.963 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:35.992 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:36.016 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:48:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:48:36.017 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:36.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:48:36.044 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:36.041+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:36.130 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:36.131 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:48:36.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:48:36.131 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [] 2024-09-15T02:48:36.131 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:36.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:48:36.145 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:36.141+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:36.246 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:36.243+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:36.252 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.252 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.261 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:48:36.347 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:36.343+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:36.357 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.357 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.366 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:48:36.448 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:36.441+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:36.466 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.466 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.476 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:48:36.530 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.10. 2024-09-15T02:48:36.545 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-09-15T02:48:36.545 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:48:36.545 DEBUG:teuthology.orchestra.run.smithi152:> type ceph-objectstore-tool 2024-09-15T02:48:36.551 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T02:48:36.552 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-09-15T02:48:36.552 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:36.546+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:36.570 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.570 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.579 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:48:36.650 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:36.645+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:36.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:48:36.650+0000 7f69db17b300 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:48:36.680 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.680 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:48:36.751 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:36.745+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:36.793 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.793 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:48:36.851 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:36.845+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:36.904 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:36.905 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:36.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:48:36.952 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:36.945+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:37.010 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:37.010 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:37.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:48:37.053 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:37.049+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:37.115 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:37.115 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:37.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:48:37.154 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:37.151+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:37.225 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:37.226 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:37.226 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:48:37.226 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:37.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:48:37.255 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:37.250+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:37.333 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:37.333 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:37.333 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:48:37.333 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:48:37.333 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:37.342 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:48:37.355 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:37.350+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:37.441 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:37.441 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:48:37.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:48:37.442 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-09-15T02:48:37.442 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:37.442 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:48:37.438+0000 7f69db17b300 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:48:37.450 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:48:37.496 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_0' created 2024-09-15T02:48:37.497 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:37.494+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:37.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:48:37.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-15T02:48:37.557 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:37.551+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:37.559 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:37.559 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:37.568 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:48:37.658 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:37.654+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:37.662 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:37.662 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:37.671 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:48:37.758 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:37.751+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:37.809 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:37.809 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:37.821 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:48:37.859 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:37.854+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:37.918 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:37.919 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:37.927 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:48:37.960 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:37.957+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:38.061 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:38.058+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:38.070 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:38.070 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:38.079 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:48:38.162 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:38.157+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:38.176 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:38.176 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:38.185 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:48:38.264 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:38.261+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:38.291 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:38.291 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:38.291 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:38.291 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:38.295 INFO:tasks.thrashosds.thrasher:No PGs found for osd.10 2024-09-15T02:48:38.295 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.10 2024-09-15T02:48:38.296 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 show-label 2024-09-15T02:48:38.300 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:48:38.330 INFO:teuthology.orchestra.run.smithi152.stdout:inferring bluefs devices from bluestore path 2024-09-15T02:48:38.330 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:38.330 INFO:teuthology.orchestra.run.smithi152.stdout: "/var/lib/ceph/osd/ceph-10/block": { 2024-09-15T02:48:38.330 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_uuid": "f446f37d-a5c3-4e59-88f9-5a0bb818b35b", 2024-09-15T02:48:38.330 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 96636764160, 2024-09-15T02:48:38.330 INFO:teuthology.orchestra.run.smithi152.stdout: "btime": "2024-09-15T02:47:41.165382+0000", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "main", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_blocks": "23592960", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_blocks_per_key": "128", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_bytes_per_block": "4096", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_size": "96636764160", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs": "1", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_fsid": "24e12693-a4e4-4f9a-bb1d-999f41be75a2", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "created_at": "2024-09-15T02:47:43.594885Z", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "kv_backend": "rocksdb", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "magic": "ceph osd volume v026", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "mkfs_done": "yes", 2024-09-15T02:48:38.331 INFO:teuthology.orchestra.run.smithi152.stdout: "ready": "ready", 2024-09-15T02:48:38.332 INFO:teuthology.orchestra.run.smithi152.stdout: "require_osd_release": "17", 2024-09-15T02:48:38.332 INFO:teuthology.orchestra.run.smithi152.stdout: "whoami": "10" 2024-09-15T02:48:38.332 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:38.332 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:38.333 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.10 2024-09-15T02:48:38.333 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 show-sharding 2024-09-15T02:48:38.365 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:38.361+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:38.390 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:38.390 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:38.400 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:48:38.466 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:38.459+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:38.495 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:38.495 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:38.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:38.496 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [] 2024-09-15T02:48:38.496 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:38.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:48:38.512 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:48:38.512 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-15T02:48:38.572 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:38.567+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:38.606 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:38.606 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:38.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:48:38.667 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:38.661+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:38.709 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:38.710 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:38.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:48:38.769 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:38.761+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:38.822 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:38.822 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:38.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:48:38.881 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:38.877+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:38.930 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:38.930 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:38.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:48:38.971 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:38.966+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:39.040 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:39.040 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:39.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:48:39.072 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:39.067+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:39.150 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:39.150 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:39.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:48:39.173 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:39.169+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:39.196 INFO:teuthology.orchestra.run.smithi152.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P 2024-09-15T02:48:39.203 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.10 2024-09-15T02:48:39.203 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-15T02:48:39.203 INFO:tasks.thrashosds.thrasher:applying new sharding: m O(5,8-) 2024-09-15T02:48:39.203 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 --sharding 'm O(5,8-)' reshard 2024-09-15T02:48:39.256 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:39.256 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:39.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:48:39.274 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:39.267+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:39.298 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:48:39.294+0000 7f9fe3e0a840 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:48:39.367 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:39.367 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:39.367 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:39.367 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:39.367 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:39.375 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:48:39.376 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:39.369+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:48:39.473 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.105974878, 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:48:39.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:48:39.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:48:39.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:48:39.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723076+0000", 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.746644685, 2024-09-15T02:48:39.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016520386000000001, 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723076+0000", 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723076+0000", 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723077+0000", 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.009e-06 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723081+0000", 2024-09-15T02:48:39.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5000000000000001e-06 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723106+0000", 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4300000000000001e-05 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723121+0000", 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5617e-05 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.732427+0000", 2024-09-15T02:48:39.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093053460000000008 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.732582+0000", 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015555699999999999 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.732799+0000", 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021657299999999999 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738098+0000", 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052989100000000004 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739475+0000", 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013773609999999999 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739546+0000", 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1459999999999997e-05 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739596+0000", 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9753000000000003e-05 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723130+0000", 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.746590873000001, 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016698528000000001, 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.486 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 49 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723130+0000", 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723130+0000", 2024-09-15T02:48:39.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723131+0000", 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.569999999999999e-07 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723137+0000", 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3230000000000003e-06 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723145+0000", 2024-09-15T02:48:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.303e-06 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723150+0000", 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9139999999999999e-06 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733435+0000", 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010284887 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733500+0000", 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5455000000000003e-05 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733600+0000", 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9709000000000004e-05 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738112+0000", 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045116660000000001 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739689+0000", 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015773779999999999 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739778+0000", 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8541000000000005e-05 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739828+0000", 2024-09-15T02:48:39.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0395000000000001e-05 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.746559002, 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016944363, 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.492 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 50 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5799999999999999e-07 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723225+0000", 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3168000000000002e-05 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723248+0000", 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3031999999999999e-05 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723255+0000", 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8889999999999996e-06 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733941+0000", 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010686161 2024-09-15T02:48:39.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.734161+0000", 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219189 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.734379+0000", 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021872800000000001 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738124+0000", 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003744561 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739946+0000", 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001821629 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740056+0000", 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011092800000000001 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740106+0000", 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9419999999999998e-05 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723270+0000", 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.746450953, 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017134052, 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.498 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 51 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723270+0000", 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723270+0000", 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723271+0000", 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.172e-06 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723278+0000", 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.746e-06 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723297+0000", 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8513999999999999e-05 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723307+0000", 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6369999999999994e-06 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.735565+0000", 2024-09-15T02:48:39.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012258118 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.735689+0000", 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012389500000000001 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.735896+0000", 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000207772 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738136+0000", 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002239343 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740202+0000", 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020667289999999998 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740288+0000", 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740404+0000", 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116017 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.746403364000001, 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.504 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:48:39.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.13781337, 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018893783000000001, 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.510 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:48:39.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331908+0000", 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.133e-06 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331914+0000", 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5799999999999999e-06 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331925+0000", 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0893999999999999e-05 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331931+0000", 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2199999999999997e-06 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342666+0000", 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01073519 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342753+0000", 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7175999999999993e-05 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342888+0000", 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013462700000000001 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348168+0000", 2024-09-15T02:48:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005280285 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350557+0000", 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023890309999999998 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350698+0000", 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140755 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350801+0000", 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102892 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.137782899999999, 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026305674000000001, 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.516 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:39.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331943+0000", 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7559999999999999e-06 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331955+0000", 2024-09-15T02:48:39.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331964+0000", 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4069999999999993e-06 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343483+0000", 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011518995000000001 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343581+0000", 2024-09-15T02:48:39.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8568999999999998e-05 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343713+0000", 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013195700000000001 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348183+0000", 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044698639999999996 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358029+0000", 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0098460219999999994 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358143+0000", 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114335 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358243+0000", 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9881000000000001e-05 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.521 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.137750222999999, 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026584729000000001, 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.522 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331971+0000", 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332135+0000", 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000164098 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332158+0000", 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332165+0000", 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8669999999999996e-06 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344120+0000", 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011955222 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344205+0000", 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4895000000000006e-05 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344336+0000", 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130844 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.349645+0000", 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053089770000000003 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358356+0000", 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087112079999999998 2024-09-15T02:48:39.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358462+0000", 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105664 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358555+0000", 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3110999999999997e-05 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.137540538, 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:48:39.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:48:39.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:48:39.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.533 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.901068698, 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017901001, 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.534 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 98 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568653+0000", 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4700000000000001e-07 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568658+0000", 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.006e-06 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568666+0000", 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5369999999999997e-06 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568671+0000", 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3700000000000003e-06 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580006+0000", 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011334946 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580064+0000", 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7207999999999998e-05 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580154+0000", 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0682999999999999e-05 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.582087+0000", 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019328520000000001 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586462+0000", 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004374452 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586503+0000", 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1872000000000002e-05 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586553+0000", 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9327999999999999e-05 2024-09-15T02:48:39.539 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.900966594, 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017954067000000001, 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.540 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 99 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568755+0000", 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9300000000000006e-07 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568847+0000", 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1906999999999999e-05 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568859+0000", 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2047e-05 2024-09-15T02:48:39.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568866+0000", 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7170000000000002e-06 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580440+0000", 2024-09-15T02:48:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011574442000000001 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580500+0000", 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9732999999999999e-05 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580611+0000", 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011137999999999999 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583173+0000", 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025617330000000001 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586614+0000", 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034406929999999999 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586671+0000", 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7398000000000002e-05 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586708+0000", 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7024000000000003e-05 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.900846919999999, 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:48:39.546 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:48:39.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:48:39.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.883328344000001, 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:48:39.552 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:39.556 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.069465917, 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:39.557 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:48:39.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:48:39.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:48:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.563 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313294+0000", 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.156426511999999, 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016588219000000001, 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-09-15T02:48:39.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313294+0000", 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313294+0000", 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313296+0000", 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9130000000000001e-06 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313485+0000", 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018899699999999999 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313507+0000", 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.226e-05 2024-09-15T02:48:39.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313520+0000", 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3027000000000001e-05 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.315971+0000", 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024512280000000002 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316058+0000", 2024-09-15T02:48:39.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6513000000000003e-05 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316201+0000", 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014322200000000001 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320589+0000", 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043881939999999998 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329755+0000", 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091656010000000007 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329815+0000", 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9849999999999998e-05 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329882+0000", 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7414000000000002e-05 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.569 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313537+0000", 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.156183701, 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016521168999999999, 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-09-15T02:48:39.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313537+0000", 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313537+0000", 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313539+0000", 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8700000000000001e-06 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313548+0000", 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4860000000000003e-06 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313592+0000", 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3717000000000002e-05 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313622+0000", 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.968e-05 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316761+0000", 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003139537 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316875+0000", 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011346200000000001 2024-09-15T02:48:39.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.317026+0000", 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015164099999999999 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320605+0000", 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003578332 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329945+0000", 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093409329999999992 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330016+0000", 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0519999999999996e-05 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330058+0000", 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1990999999999998e-05 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.575 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313636+0000", 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.156084061, 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016775641000000001, 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.576 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313636+0000", 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313636+0000", 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313638+0000", 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.57e-06 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313647+0000", 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2529999999999993e-06 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313665+0000", 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8003e-05 2024-09-15T02:48:39.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313691+0000", 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5264000000000001e-05 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.317796+0000", 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041053410000000002 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.317908+0000", 2024-09-15T02:48:39.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111722 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.318100+0000", 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019205099999999999 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320618+0000", 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002518831 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330209+0000", 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095900470000000008 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330342+0000", 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013296599999999999 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330412+0000", 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0593000000000007e-05 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313720+0000", 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.156000670999999, 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016893389000000002, 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.582 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313720+0000", 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313720+0000", 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313721+0000", 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.406e-06 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313870+0000", 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014921800000000001 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313897+0000", 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6812999999999999e-05 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313948+0000", 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000001e-05 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.318791+0000", 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048430310000000002 2024-09-15T02:48:39.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.318931+0000", 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000139603 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.319168+0000", 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023724600000000001 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.321596+0000", 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024279729999999999 2024-09-15T02:48:39.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330505+0000", 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0089086970000000001 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330558+0000", 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3513000000000001e-05 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330613+0000", 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5179000000000001e-05 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:142 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313967+0000", 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.15575344, 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017001057999999999, 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:39.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 142 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313967+0000", 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313967+0000", 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313969+0000", 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.55e-06 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313979+0000", 2024-09-15T02:48:39.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0603e-05 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313997+0000", 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7629999999999999e-05 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.314012+0000", 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4756999999999999e-05 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:39.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320653+0000", 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066416599999999998 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320783+0000", 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012967299999999999 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320967+0000", 2024-09-15T02:48:39.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000183821 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.322180+0000", 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012135830000000001 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330877+0000", 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0086965759999999993 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330925+0000", 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8513999999999997e-05 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330968+0000", 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2691000000000002e-05 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:39.594 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:39.595 DEBUG:teuthology.orchestra.run.smithi191:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-15T02:48:39.596 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:39.590+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:39.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:48:39.615 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight read -> 100 2024-09-15T02:48:39.615 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight delete -> 50 2024-09-15T02:48:39.615 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight snap_create -> 50 2024-09-15T02:48:39.615 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight snap_remove -> 50 2024-09-15T02:48:39.615 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight rollback -> 50 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight setattr -> 25 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight rmattr -> 25 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight copy_from -> 50 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout:adding op weight append -> 100 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout:ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable) 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout:Configuration: 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout: Number of operations: 400 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout: Number of objects: 50 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout: Max in flight operations: 16 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout: Object size (in bytes): 4000000 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout: Write stride min: 400000 2024-09-15T02:48:39.616 INFO:tasks.rados.rados.0.smithi191.stdout: Write stride max: 800000 2024-09-15T02:48:39.686 INFO:teuthology.orchestra.run.smithi152.stdout:reshard success 2024-09-15T02:48:39.696 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:39.693+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:39.705 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:39.705 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:39.705 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:39.705 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:39.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:48:39.797 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:39.793+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:39.859 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:39.860 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:39.860 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:39.860 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:39.860 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:39.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:48:39.898 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:39.891+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:39.905 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:39.931 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:39.960 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:39.967 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:39.967 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:39.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:39.967 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-09-15T02:48:39.967 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:39.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:48:39.993 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:39.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:39.987+0000 7f8a23fff700 0 -- 172.21.15.6:0/1463960677 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7f8a24193910 msgr2=0x7f8a24195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:48:39.999 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:39.991+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:40.078 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.078 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.087 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.10 2024-09-15T02:48:40.087 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 fsck 2024-09-15T02:48:40.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:48:40.100 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:40.093+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:40.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:48:40.134+0000 7f9a8fd33840 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:48:40.200 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:40.195+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:40.227 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:48:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:48:40.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:48:40.238 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.238 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:48:40.302 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:40.297+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:40.346 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.346 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:48:40.403 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:40.395+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:40.459 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.460 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:48:40.504 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:40.495+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:40.515 INFO:tasks.rados.rados.0.smithi191.stdout:1: write initial oid 1 2024-09-15T02:48:40.515 INFO:tasks.rados.rados.0.smithi191.stdout:1: seq_num 0 ranges {0=32768} 2024-09-15T02:48:40.515 INFO:tasks.rados.rados.0.smithi191.stdout:1: writing smithi19125844-1 from 0 to 32768 tid 1 2024-09-15T02:48:40.516 INFO:tasks.rados.rados.0.smithi191.stdout:2: write initial oid 2 2024-09-15T02:48:40.516 INFO:tasks.rados.rados.0.smithi191.stdout:2: seq_num 1 ranges {0=32768} 2024-09-15T02:48:40.516 INFO:tasks.rados.rados.0.smithi191.stdout:2: writing smithi19125844-2 from 0 to 32768 tid 1 2024-09-15T02:48:40.516 INFO:tasks.rados.rados.0.smithi191.stdout:3: write initial oid 3 2024-09-15T02:48:40.516 INFO:tasks.rados.rados.0.smithi191.stdout:3: seq_num 2 ranges {0=49152} 2024-09-15T02:48:40.517 INFO:tasks.rados.rados.0.smithi191.stdout:3: writing smithi19125844-3 from 0 to 49152 tid 1 2024-09-15T02:48:40.517 INFO:tasks.rados.rados.0.smithi191.stdout:4: write initial oid 4 2024-09-15T02:48:40.517 INFO:tasks.rados.rados.0.smithi191.stdout:4: seq_num 3 ranges {0=65536} 2024-09-15T02:48:40.519 INFO:tasks.rados.rados.0.smithi191.stdout:4: writing smithi19125844-4 from 0 to 65536 tid 1 2024-09-15T02:48:40.519 INFO:tasks.rados.rados.0.smithi191.stdout:5: write initial oid 5 2024-09-15T02:48:40.519 INFO:tasks.rados.rados.0.smithi191.stdout:5: seq_num 4 ranges {0=81920} 2024-09-15T02:48:40.520 INFO:tasks.rados.rados.0.smithi191.stdout:5: writing smithi19125844-5 from 0 to 81920 tid 1 2024-09-15T02:48:40.521 INFO:tasks.rados.rados.0.smithi191.stdout:6: write initial oid 6 2024-09-15T02:48:40.521 INFO:tasks.rados.rados.0.smithi191.stdout:6: seq_num 5 ranges {0=98304} 2024-09-15T02:48:40.522 INFO:tasks.rados.rados.0.smithi191.stdout:6: writing smithi19125844-6 from 0 to 98304 tid 1 2024-09-15T02:48:40.522 INFO:tasks.rados.rados.0.smithi191.stdout:7: write initial oid 7 2024-09-15T02:48:40.523 INFO:tasks.rados.rados.0.smithi191.stdout:7: seq_num 6 ranges {0=114688} 2024-09-15T02:48:40.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:48:40.526 INFO:tasks.rados.rados.0.smithi191.stdout:7: writing smithi19125844-7 from 0 to 114688 tid 1 2024-09-15T02:48:40.526 INFO:tasks.rados.rados.0.smithi191.stdout:8: write initial oid 8 2024-09-15T02:48:40.526 INFO:tasks.rados.rados.0.smithi191.stdout:8: seq_num 7 ranges {0=131072} 2024-09-15T02:48:40.527 INFO:tasks.rados.rados.0.smithi191.stdout:8: writing smithi19125844-8 from 0 to 131072 tid 1 2024-09-15T02:48:40.527 INFO:tasks.rados.rados.0.smithi191.stdout:9: write initial oid 9 2024-09-15T02:48:40.527 INFO:tasks.rados.rados.0.smithi191.stdout:9: seq_num 8 ranges {0=147456} 2024-09-15T02:48:40.528 INFO:tasks.rados.rados.0.smithi191.stdout:9: writing smithi19125844-9 from 0 to 147456 tid 1 2024-09-15T02:48:40.528 INFO:tasks.rados.rados.0.smithi191.stdout:10: write initial oid 10 2024-09-15T02:48:40.528 INFO:tasks.rados.rados.0.smithi191.stdout:10: seq_num 9 ranges {0=163840} 2024-09-15T02:48:40.529 INFO:tasks.rados.rados.0.smithi191.stdout:10: writing smithi19125844-10 from 0 to 163840 tid 1 2024-09-15T02:48:40.529 INFO:tasks.rados.rados.0.smithi191.stdout:11: write initial oid 11 2024-09-15T02:48:40.530 INFO:tasks.rados.rados.0.smithi191.stdout:11: seq_num 10 ranges {0=180224} 2024-09-15T02:48:40.531 INFO:tasks.rados.rados.0.smithi191.stdout:11: writing smithi19125844-11 from 0 to 180224 tid 1 2024-09-15T02:48:40.531 INFO:tasks.rados.rados.0.smithi191.stdout:12: write initial oid 12 2024-09-15T02:48:40.531 INFO:tasks.rados.rados.0.smithi191.stdout:12: seq_num 11 ranges {0=196608} 2024-09-15T02:48:40.533 INFO:tasks.rados.rados.0.smithi191.stdout:12: writing smithi19125844-12 from 0 to 196608 tid 1 2024-09-15T02:48:40.533 INFO:tasks.rados.rados.0.smithi191.stdout:13: write initial oid 13 2024-09-15T02:48:40.533 INFO:tasks.rados.rados.0.smithi191.stdout:13: seq_num 12 ranges {0=212992} 2024-09-15T02:48:40.534 INFO:tasks.rados.rados.0.smithi191.stdout:13: writing smithi19125844-13 from 0 to 212992 tid 1 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:1: finishing write tid 1 to smithi19125844-1 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:2: finishing write tid 1 to smithi19125844-2 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:2: finishing write tid 2 to smithi19125844-2 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:3: finishing write tid 1 to smithi19125844-3 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:3: finishing write tid 2 to smithi19125844-3 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:14: write initial oid 14 2024-09-15T02:48:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:14: seq_num 13 ranges {0=229376} 2024-09-15T02:48:40.536 INFO:tasks.rados.rados.0.smithi191.stdout:14: writing smithi19125844-14 from 0 to 229376 tid 1 2024-09-15T02:48:40.537 INFO:tasks.rados.rados.0.smithi191.stdout:4: finishing write tid 1 to smithi19125844-4 2024-09-15T02:48:40.537 INFO:tasks.rados.rados.0.smithi191.stdout:15: write initial oid 15 2024-09-15T02:48:40.537 INFO:tasks.rados.rados.0.smithi191.stdout:15: seq_num 14 ranges {0=245760} 2024-09-15T02:48:40.538 INFO:tasks.rados.rados.0.smithi191.stdout:15: writing smithi19125844-15 from 0 to 245760 tid 1 2024-09-15T02:48:40.538 INFO:tasks.rados.rados.0.smithi191.stdout:16: write initial oid 16 2024-09-15T02:48:40.538 INFO:tasks.rados.rados.0.smithi191.stdout:16: seq_num 15 ranges {0=262144} 2024-09-15T02:48:40.540 INFO:tasks.rados.rados.0.smithi191.stdout:16: writing smithi19125844-16 from 0 to 262144 tid 1 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:1: finishing write tid 2 to smithi19125844-1 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:4: finishing write tid 2 to smithi19125844-4 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:2: finishing write tid 3 to smithi19125844-2 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:2: done (15 left) 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:17: write initial oid 17 2024-09-15T02:48:40.541 INFO:tasks.rados.rados.0.smithi191.stdout:17: seq_num 16 ranges {0=278528} 2024-09-15T02:48:40.546 INFO:tasks.rados.rados.0.smithi191.stdout:17: writing smithi19125844-17 from 0 to 278528 tid 1 2024-09-15T02:48:40.546 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.546 INFO:tasks.rados.rados.0.smithi191.stdout:3: finishing write tid 3 to smithi19125844-3 2024-09-15T02:48:40.546 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-15T02:48:40.546 INFO:tasks.rados.rados.0.smithi191.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-15T02:48:40.547 INFO:tasks.rados.rados.0.smithi191.stdout:3: done (15 left) 2024-09-15T02:48:40.547 INFO:tasks.rados.rados.0.smithi191.stdout:18: write initial oid 18 2024-09-15T02:48:40.547 INFO:tasks.rados.rados.0.smithi191.stdout:18: seq_num 17 ranges {0=294912} 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:18: writing smithi19125844-18 from 0 to 294912 tid 1 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:5: finishing write tid 1 to smithi19125844-5 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:5: finishing write tid 2 to smithi19125844-5 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:4: finishing write tid 3 to smithi19125844-4 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:4: done (15 left) 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:19: write initial oid 19 2024-09-15T02:48:40.549 INFO:tasks.rados.rados.0.smithi191.stdout:19: seq_num 18 ranges {0=311296} 2024-09-15T02:48:40.551 INFO:tasks.rados.rados.0.smithi191.stdout:19: writing smithi19125844-19 from 0 to 311296 tid 1 2024-09-15T02:48:40.551 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.551 INFO:tasks.rados.rados.0.smithi191.stdout:11: finishing write tid 1 to smithi19125844-11 2024-09-15T02:48:40.551 INFO:tasks.rados.rados.0.smithi191.stdout:13: finishing write tid 1 to smithi19125844-13 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:13: finishing write tid 2 to smithi19125844-13 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:1: finishing write tid 3 to smithi19125844-1 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:15: finishing write tid 1 to smithi19125844-15 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:5: finishing write tid 3 to smithi19125844-5 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 5 v 4 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:1: done (15 left) 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:5: done (14 left) 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:20: write initial oid 20 2024-09-15T02:48:40.552 INFO:tasks.rados.rados.0.smithi191.stdout:20: seq_num 19 ranges {0=327680} 2024-09-15T02:48:40.553 INFO:tasks.rados.rados.0.smithi191.stdout:20: writing smithi19125844-20 from 0 to 327680 tid 1 2024-09-15T02:48:40.554 INFO:tasks.rados.rados.0.smithi191.stdout:8: finishing write tid 1 to smithi19125844-8 2024-09-15T02:48:40.554 INFO:tasks.rados.rados.0.smithi191.stdout:11: finishing write tid 2 to smithi19125844-11 2024-09-15T02:48:40.554 INFO:tasks.rados.rados.0.smithi191.stdout:6: finishing write tid 1 to smithi19125844-6 2024-09-15T02:48:40.554 INFO:tasks.rados.rados.0.smithi191.stdout:6: finishing write tid 2 to smithi19125844-6 2024-09-15T02:48:40.554 INFO:tasks.rados.rados.0.smithi191.stdout:8: finishing write tid 2 to smithi19125844-8 2024-09-15T02:48:40.554 INFO:tasks.rados.rados.0.smithi191.stdout:21: write initial oid 21 2024-09-15T02:48:40.555 INFO:tasks.rados.rados.0.smithi191.stdout:21: seq_num 20 ranges {0=344064} 2024-09-15T02:48:40.556 INFO:tasks.rados.rados.0.smithi191.stdout:21: writing smithi19125844-21 from 0 to 344064 tid 1 2024-09-15T02:48:40.557 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:6: finishing write tid 3 to smithi19125844-6 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:13: finishing write tid 3 to smithi19125844-13 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:6: done (15 left) 2024-09-15T02:48:40.558 INFO:tasks.rados.rados.0.smithi191.stdout:13: done (14 left) 2024-09-15T02:48:40.559 INFO:tasks.rados.rados.0.smithi191.stdout:22: write initial oid 22 2024-09-15T02:48:40.561 INFO:tasks.rados.rados.0.smithi191.stdout:22: seq_num 21 ranges {0=360448} 2024-09-15T02:48:40.564 INFO:tasks.rados.rados.0.smithi191.stdout:22: writing smithi19125844-22 from 0 to 360448 tid 1 2024-09-15T02:48:40.564 INFO:tasks.rados.rados.0.smithi191.stdout:23: write initial oid 23 2024-09-15T02:48:40.564 INFO:tasks.rados.rados.0.smithi191.stdout:23: seq_num 22 ranges {0=376832} 2024-09-15T02:48:40.567 INFO:tasks.rados.rados.0.smithi191.stdout:23: writing smithi19125844-23 from 0 to 376832 tid 1 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:11: finishing write tid 3 to smithi19125844-11 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:10: finishing write tid 1 to smithi19125844-10 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:14: finishing write tid 1 to smithi19125844-14 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:14: finishing write tid 2 to smithi19125844-14 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:15: finishing write tid 2 to smithi19125844-15 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:10: finishing write tid 2 to smithi19125844-10 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:12: finishing write tid 1 to smithi19125844-12 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:12: finishing write tid 2 to smithi19125844-12 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:9: finishing write tid 1 to smithi19125844-9 2024-09-15T02:48:40.568 INFO:tasks.rados.rados.0.smithi191.stdout:9: finishing write tid 2 to smithi19125844-9 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:8: finishing write tid 3 to smithi19125844-8 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:16: finishing write tid 1 to smithi19125844-16 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:7: finishing write tid 1 to smithi19125844-7 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:7: finishing write tid 2 to smithi19125844-7 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:17: finishing write tid 1 to smithi19125844-17 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:8: done (15 left) 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:11: done (14 left) 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:24: write initial oid 24 2024-09-15T02:48:40.569 INFO:tasks.rados.rados.0.smithi191.stdout:24: seq_num 23 ranges {0=393216} 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:24: writing smithi19125844-24 from 0 to 393216 tid 1 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:19: finishing write tid 1 to smithi19125844-19 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:15: finishing write tid 3 to smithi19125844-15 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:19: finishing write tid 2 to smithi19125844-19 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:16: finishing write tid 2 to smithi19125844-16 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:10: finishing write tid 3 to smithi19125844-10 2024-09-15T02:48:40.571 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:12: finishing write tid 3 to smithi19125844-12 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:18: finishing write tid 1 to smithi19125844-18 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:18: finishing write tid 2 to smithi19125844-18 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:7: finishing write tid 3 to smithi19125844-7 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:7: done (14 left) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:10: done (13 left) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:12: done (12 left) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:15: done (11 left) 2024-09-15T02:48:40.572 INFO:tasks.rados.rados.0.smithi191.stdout:25: write initial oid 25 2024-09-15T02:48:40.573 INFO:tasks.rados.rados.0.smithi191.stdout:25: seq_num 24 ranges {0=409600} 2024-09-15T02:48:40.574 INFO:tasks.rados.rados.0.smithi191.stdout:25: writing smithi19125844-25 from 0 to 409600 tid 1 2024-09-15T02:48:40.574 INFO:tasks.rados.rados.0.smithi191.stdout:26: write initial oid 26 2024-09-15T02:48:40.575 INFO:tasks.rados.rados.0.smithi191.stdout:26: seq_num 25 ranges {0=425984} 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:26: writing smithi19125844-26 from 0 to 425984 tid 1 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:21: finishing write tid 1 to smithi19125844-21 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:21: finishing write tid 2 to smithi19125844-21 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:16: finishing write tid 3 to smithi19125844-16 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:18: finishing write tid 3 to smithi19125844-18 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 18 v 2 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:14: finishing write tid 3 to smithi19125844-14 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-15T02:48:40.578 INFO:tasks.rados.rados.0.smithi191.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:20: finishing write tid 1 to smithi19125844-20 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:20: finishing write tid 2 to smithi19125844-20 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:19: finishing write tid 3 to smithi19125844-19 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:9: finishing write tid 3 to smithi19125844-9 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 9 v 8 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:9: done (12 left) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:14: done (11 left) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:16: done (10 left) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:18: done (9 left) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:19: done (8 left) 2024-09-15T02:48:40.579 INFO:tasks.rados.rados.0.smithi191.stdout:27: write initial oid 27 2024-09-15T02:48:40.580 INFO:tasks.rados.rados.0.smithi191.stdout:27: seq_num 26 ranges {0=442368} 2024-09-15T02:48:40.580 INFO:tasks.rados.rados.0.smithi191.stdout:27: writing smithi19125844-27 from 0 to 442368 tid 1 2024-09-15T02:48:40.580 INFO:tasks.rados.rados.0.smithi191.stdout:28: write initial oid 28 2024-09-15T02:48:40.580 INFO:tasks.rados.rados.0.smithi191.stdout:28: seq_num 27 ranges {0=458752} 2024-09-15T02:48:40.583 INFO:tasks.rados.rados.0.smithi191.stdout:28: writing smithi19125844-28 from 0 to 458752 tid 1 2024-09-15T02:48:40.583 INFO:tasks.rados.rados.0.smithi191.stdout:29: write initial oid 29 2024-09-15T02:48:40.583 INFO:tasks.rados.rados.0.smithi191.stdout:29: seq_num 28 ranges {0=475136} 2024-09-15T02:48:40.586 INFO:tasks.rados.rados.0.smithi191.stdout:29: writing smithi19125844-29 from 0 to 475136 tid 1 2024-09-15T02:48:40.586 INFO:tasks.rados.rados.0.smithi191.stdout:30: write initial oid 30 2024-09-15T02:48:40.586 INFO:tasks.rados.rados.0.smithi191.stdout:30: seq_num 29 ranges {0=491520} 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:30: writing smithi19125844-30 from 0 to 491520 tid 1 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:21: finishing write tid 3 to smithi19125844-21 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:20: finishing write tid 3 to smithi19125844-20 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:22: finishing write tid 1 to smithi19125844-22 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:25: finishing write tid 1 to smithi19125844-25 2024-09-15T02:48:40.590 INFO:tasks.rados.rados.0.smithi191.stdout:25: finishing write tid 2 to smithi19125844-25 2024-09-15T02:48:40.591 INFO:tasks.rados.rados.0.smithi191.stdout:22: finishing write tid 2 to smithi19125844-22 2024-09-15T02:48:40.591 INFO:tasks.rados.rados.0.smithi191.stdout:20: done (11 left) 2024-09-15T02:48:40.591 INFO:tasks.rados.rados.0.smithi191.stdout:21: done (10 left) 2024-09-15T02:48:40.591 INFO:tasks.rados.rados.0.smithi191.stdout:31: write initial oid 31 2024-09-15T02:48:40.591 INFO:tasks.rados.rados.0.smithi191.stdout:31: seq_num 30 ranges {0=507904} 2024-09-15T02:48:40.593 INFO:tasks.rados.rados.0.smithi191.stdout:31: writing smithi19125844-31 from 0 to 507904 tid 1 2024-09-15T02:48:40.593 INFO:tasks.rados.rados.0.smithi191.stdout:32: write initial oid 32 2024-09-15T02:48:40.593 INFO:tasks.rados.rados.0.smithi191.stdout:32: seq_num 31 ranges {0=524288} 2024-09-15T02:48:40.596 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.597 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.597 INFO:tasks.rados.rados.0.smithi191.stdout:32: writing smithi19125844-32 from 0 to 524288 tid 1 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:25: finishing write tid 3 to smithi19125844-25 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:24: finishing write tid 1 to smithi19125844-24 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:24: finishing write tid 2 to smithi19125844-24 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:22: finishing write tid 3 to smithi19125844-22 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 22 v 2 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:24: finishing write tid 3 to smithi19125844-24 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:22: done (11 left) 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:24: done (10 left) 2024-09-15T02:48:40.598 INFO:tasks.rados.rados.0.smithi191.stdout:25: done (9 left) 2024-09-15T02:48:40.599 INFO:tasks.rados.rados.0.smithi191.stdout:33: write initial oid 33 2024-09-15T02:48:40.599 INFO:tasks.rados.rados.0.smithi191.stdout:33: seq_num 32 ranges {0=540672} 2024-09-15T02:48:40.601 INFO:tasks.rados.rados.0.smithi191.stdout:33: writing smithi19125844-33 from 0 to 540672 tid 1 2024-09-15T02:48:40.601 INFO:tasks.rados.rados.0.smithi191.stdout:34: write initial oid 34 2024-09-15T02:48:40.601 INFO:tasks.rados.rados.0.smithi191.stdout:34: seq_num 33 ranges {0=557056} 2024-09-15T02:48:40.604 INFO:tasks.rados.rados.0.smithi191.stdout:34: writing smithi19125844-34 from 0 to 557056 tid 1 2024-09-15T02:48:40.604 INFO:tasks.rados.rados.0.smithi191.stdout:35: write initial oid 35 2024-09-15T02:48:40.604 INFO:tasks.rados.rados.0.smithi191.stdout:35: seq_num 34 ranges {0=573440} 2024-09-15T02:48:40.605 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:40.597+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:40.608 INFO:tasks.rados.rados.0.smithi191.stdout:35: writing smithi19125844-35 from 0 to 573440 tid 1 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:27: finishing write tid 1 to smithi19125844-27 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:29: finishing write tid 1 to smithi19125844-29 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:27: finishing write tid 2 to smithi19125844-27 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:27: finishing write tid 3 to smithi19125844-27 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:29: finishing write tid 2 to smithi19125844-29 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:26: finishing write tid 1 to smithi19125844-26 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:26: finishing write tid 2 to smithi19125844-26 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:29: finishing write tid 3 to smithi19125844-29 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 29 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:27: done (11 left) 2024-09-15T02:48:40.609 INFO:tasks.rados.rados.0.smithi191.stdout:29: done (10 left) 2024-09-15T02:48:40.610 INFO:tasks.rados.rados.0.smithi191.stdout:36: write initial oid 36 2024-09-15T02:48:40.610 INFO:tasks.rados.rados.0.smithi191.stdout:36: seq_num 35 ranges {0=589824} 2024-09-15T02:48:40.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:48:40.612 INFO:tasks.rados.rados.0.smithi191.stdout:36: writing smithi19125844-36 from 0 to 589824 tid 1 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:31: finishing write tid 1 to smithi19125844-31 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:26: finishing write tid 3 to smithi19125844-26 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:31: finishing write tid 2 to smithi19125844-31 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:26: done (10 left) 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:37: write initial oid 37 2024-09-15T02:48:40.613 INFO:tasks.rados.rados.0.smithi191.stdout:37: seq_num 36 ranges {0=606208} 2024-09-15T02:48:40.616 INFO:tasks.rados.rados.0.smithi191.stdout:37: writing smithi19125844-37 from 0 to 606208 tid 1 2024-09-15T02:48:40.616 INFO:tasks.rados.rados.0.smithi191.stdout:38: write initial oid 38 2024-09-15T02:48:40.616 INFO:tasks.rados.rados.0.smithi191.stdout:38: seq_num 37 ranges {0=622592} 2024-09-15T02:48:40.619 INFO:tasks.rados.rados.0.smithi191.stdout:38: writing smithi19125844-38 from 0 to 622592 tid 1 2024-09-15T02:48:40.620 INFO:tasks.rados.rados.0.smithi191.stdout:39: write initial oid 39 2024-09-15T02:48:40.620 INFO:tasks.rados.rados.0.smithi191.stdout:39: seq_num 38 ranges {0=638976} 2024-09-15T02:48:40.624 INFO:tasks.rados.rados.0.smithi191.stdout:39: writing smithi19125844-39 from 0 to 638976 tid 1 2024-09-15T02:48:40.624 INFO:tasks.rados.rados.0.smithi191.stdout:40: write initial oid 40 2024-09-15T02:48:40.624 INFO:tasks.rados.rados.0.smithi191.stdout:40: seq_num 39 ranges {0=671744} 2024-09-15T02:48:40.628 INFO:tasks.rados.rados.0.smithi191.stdout:40: writing smithi19125844-40 from 0 to 671744 tid 1 2024-09-15T02:48:40.628 INFO:tasks.rados.rados.0.smithi191.stdout:41: write initial oid 41 2024-09-15T02:48:40.628 INFO:tasks.rados.rados.0.smithi191.stdout:41: seq_num 40 ranges {0=688128} 2024-09-15T02:48:40.633 INFO:tasks.rados.rados.0.smithi191.stdout:41: writing smithi19125844-41 from 0 to 688128 tid 1 2024-09-15T02:48:40.633 INFO:tasks.rados.rados.0.smithi191.stdout:42: write initial oid 42 2024-09-15T02:48:40.633 INFO:tasks.rados.rados.0.smithi191.stdout:42: seq_num 41 ranges {0=704512} 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:42: writing smithi19125844-42 from 0 to 704512 tid 1 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:30: finishing write tid 1 to smithi19125844-30 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:23: finishing write tid 1 to smithi19125844-23 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:23: finishing write tid 2 to smithi19125844-23 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:36: finishing write tid 1 to smithi19125844-36 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:36: finishing write tid 2 to smithi19125844-36 2024-09-15T02:48:40.641 INFO:tasks.rados.rados.0.smithi191.stdout:31: finishing write tid 3 to smithi19125844-31 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:36: finishing write tid 3 to smithi19125844-36 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:33: finishing write tid 1 to smithi19125844-33 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:33: finishing write tid 2 to smithi19125844-33 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:23: finishing write tid 3 to smithi19125844-23 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:33: finishing write tid 3 to smithi19125844-33 2024-09-15T02:48:40.642 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:37: finishing write tid 1 to smithi19125844-37 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:37: finishing write tid 2 to smithi19125844-37 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:37: finishing write tid 3 to smithi19125844-37 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:32: finishing write tid 1 to smithi19125844-32 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:23: done (15 left) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:31: done (14 left) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:33: done (13 left) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:36: done (12 left) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:37: done (11 left) 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:43: write initial oid 43 2024-09-15T02:48:40.643 INFO:tasks.rados.rados.0.smithi191.stdout:43: seq_num 42 ranges {0=720896} 2024-09-15T02:48:40.644 INFO:tasks.rados.rados.0.smithi191.stdout:43: writing smithi19125844-43 from 0 to 720896 tid 1 2024-09-15T02:48:40.644 INFO:tasks.rados.rados.0.smithi191.stdout:44: write initial oid 44 2024-09-15T02:48:40.644 INFO:tasks.rados.rados.0.smithi191.stdout:44: seq_num 43 ranges {0=737280} 2024-09-15T02:48:40.647 INFO:tasks.rados.rados.0.smithi191.stdout:44: writing smithi19125844-44 from 0 to 737280 tid 1 2024-09-15T02:48:40.647 INFO:tasks.rados.rados.0.smithi191.stdout:45: write initial oid 45 2024-09-15T02:48:40.647 INFO:tasks.rados.rados.0.smithi191.stdout:45: seq_num 44 ranges {0=753664} 2024-09-15T02:48:40.653 INFO:tasks.rados.rados.0.smithi191.stdout:45: writing smithi19125844-45 from 0 to 753664 tid 1 2024-09-15T02:48:40.653 INFO:tasks.rados.rados.0.smithi191.stdout:46: write initial oid 46 2024-09-15T02:48:40.653 INFO:tasks.rados.rados.0.smithi191.stdout:46: seq_num 45 ranges {0=770048} 2024-09-15T02:48:40.658 INFO:tasks.rados.rados.0.smithi191.stdout:46: writing smithi19125844-46 from 0 to 770048 tid 1 2024-09-15T02:48:40.658 INFO:tasks.rados.rados.0.smithi191.stdout:47: write initial oid 47 2024-09-15T02:48:40.658 INFO:tasks.rados.rados.0.smithi191.stdout:47: seq_num 46 ranges {0=786432} 2024-09-15T02:48:40.664 INFO:tasks.rados.rados.0.smithi191.stdout:47: writing smithi19125844-47 from 0 to 786432 tid 1 2024-09-15T02:48:40.664 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.664 INFO:tasks.rados.rados.0.smithi191.stdout:38: finishing write tid 1 to smithi19125844-38 2024-09-15T02:48:40.664 INFO:tasks.rados.rados.0.smithi191.stdout:38: finishing write tid 2 to smithi19125844-38 2024-09-15T02:48:40.664 INFO:tasks.rados.rados.0.smithi191.stdout:39: finishing write tid 1 to smithi19125844-39 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:39: finishing write tid 2 to smithi19125844-39 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:38: finishing write tid 3 to smithi19125844-38 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:40: finishing write tid 1 to smithi19125844-40 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:40: finishing write tid 2 to smithi19125844-40 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:39: finishing write tid 3 to smithi19125844-39 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:41: finishing write tid 1 to smithi19125844-41 2024-09-15T02:48:40.665 INFO:tasks.rados.rados.0.smithi191.stdout:32: finishing write tid 2 to smithi19125844-32 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:32: finishing write tid 3 to smithi19125844-32 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 32 v 8 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:32: done (15 left) 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:38: done (14 left) 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:39: done (13 left) 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:48: write initial oid 48 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-15T02:48:40.666 INFO:tasks.rados.rados.0.smithi191.stdout:48: writing smithi19125844-48 from 0 to 425984 tid 1 2024-09-15T02:48:40.669 INFO:tasks.rados.rados.0.smithi191.stdout:48: writing smithi19125844-48 from 425984 to 802816 tid 2 2024-09-15T02:48:40.669 INFO:tasks.rados.rados.0.smithi191.stdout:49: write initial oid 49 2024-09-15T02:48:40.669 INFO:tasks.rados.rados.0.smithi191.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-15T02:48:40.671 INFO:tasks.rados.rados.0.smithi191.stdout:49: writing smithi19125844-49 from 0 to 442368 tid 1 2024-09-15T02:48:40.674 INFO:tasks.rados.rados.0.smithi191.stdout:49: writing smithi19125844-49 from 442368 to 819200 tid 2 2024-09-15T02:48:40.674 INFO:tasks.rados.rados.0.smithi191.stdout:50: write initial oid 50 2024-09-15T02:48:40.674 INFO:tasks.rados.rados.0.smithi191.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-15T02:48:40.677 INFO:tasks.rados.rados.0.smithi191.stdout:50: writing smithi19125844-50 from 0 to 458752 tid 1 2024-09-15T02:48:40.681 INFO:tasks.rados.rados.0.smithi191.stdout:50: writing smithi19125844-50 from 458752 to 835584 tid 2 2024-09-15T02:48:40.681 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.681 INFO:tasks.rados.rados.0.smithi191.stdout:43: finishing write tid 1 to smithi19125844-43 2024-09-15T02:48:40.681 INFO:tasks.rados.rados.0.smithi191.stdout:44: finishing write tid 1 to smithi19125844-44 2024-09-15T02:48:40.681 INFO:tasks.rados.rados.0.smithi191.stdout:44: finishing write tid 2 to smithi19125844-44 2024-09-15T02:48:40.681 INFO:tasks.rados.rados.0.smithi191.stdout:45: finishing write tid 1 to smithi19125844-45 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:45: finishing write tid 2 to smithi19125844-45 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:45: finishing write tid 3 to smithi19125844-45 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:41: finishing write tid 2 to smithi19125844-41 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:45: done (15 left) 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:51: rmattr oid 18 current snap is 0 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:51: done (15 left) 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:52: copy_from oid 38 from oid 26 current snap is 0 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:44: finishing write tid 3 to smithi19125844-44 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 44 v 8 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-15T02:48:40.682 INFO:tasks.rados.rados.0.smithi191.stdout:43: finishing write tid 2 to smithi19125844-43 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:48: finishing write tid 1 to smithi19125844-48 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:46: finishing write tid 1 to smithi19125844-46 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:46: finishing write tid 2 to smithi19125844-46 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:43: finishing write tid 3 to smithi19125844-43 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:41: finishing write tid 3 to smithi19125844-41 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 41 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:41: done (15 left) 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:43: done (14 left) 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:44: done (13 left) 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:53: copy_from oid 37 from oid 4 current snap is 0 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:54: read oid 45 snap -1 2024-09-15T02:48:40.683 INFO:tasks.rados.rados.0.smithi191.stdout:54: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-15T02:48:40.684 INFO:tasks.rados.rados.0.smithi191.stdout:55: copy_from oid 41 from oid 43 current snap is 0 2024-09-15T02:48:40.684 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.689 INFO:tasks.rados.rados.0.smithi191.stdout:48: finishing write tid 2 to smithi19125844-48 2024-09-15T02:48:40.689 INFO:tasks.rados.rados.0.smithi191.stdout:48: finishing write tid 3 to smithi19125844-48 2024-09-15T02:48:40.689 INFO:tasks.rados.rados.0.smithi191.stdout:54: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-15T02:48:40.705 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:40.701+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:48: finishing write tid 4 to smithi19125844-48 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 5 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:17: finishing write tid 2 to smithi19125844-17 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:50: finishing write tid 1 to smithi19125844-50 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:17: finishing write tid 3 to smithi19125844-17 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:50: finishing write tid 2 to smithi19125844-50 2024-09-15T02:48:40.710 INFO:tasks.rados.rados.0.smithi191.stdout:52: finishing copy_from racing read to smithi19125844-38 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:52: finishing copy_from to smithi19125844-38 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 38 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:17: done (15 left) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:48: done (14 left) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:52: done (13 left) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:54: done (12 left) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:56: rmattr oid 8 current snap is 0 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:56: done (12 left) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:57: read oid 15 snap -1 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:57: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:58: rollback oid 12 current snap is 0 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:58: done (13 left) 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:59: rollback oid 19 current snap is 0 2024-09-15T02:48:40.711 INFO:tasks.rados.rados.0.smithi191.stdout:59: done (13 left) 2024-09-15T02:48:40.712 INFO:tasks.rados.rados.0.smithi191.stdout:60: copy_from oid 2 from oid 44 current snap is 0 2024-09-15T02:48:40.712 INFO:tasks.rados.rados.0.smithi191.stdout:61: copy_from oid 33 from oid 24 current snap is 0 2024-09-15T02:48:40.712 INFO:tasks.rados.rados.0.smithi191.stdout:62: rollback oid 29 current snap is 0 2024-09-15T02:48:40.712 INFO:tasks.rados.rados.0.smithi191.stdout:62: done (15 left) 2024-09-15T02:48:40.712 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 5 current snap is 0 2024-09-15T02:48:40.712 INFO:tasks.rados.rados.0.smithi191.stdout:63: seq_num 50 ranges {81920=475136,557056=376832} 2024-09-15T02:48:40.715 INFO:tasks.rados.rados.0.smithi191.stdout:63: writing smithi19125844-5 from 81920 to 557056 tid 1 2024-09-15T02:48:40.720 INFO:tasks.rados.rados.0.smithi191.stdout:63: writing smithi19125844-5 from 557056 to 933888 tid 2 2024-09-15T02:48:40.720 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.720 INFO:tasks.rados.rados.0.smithi191.stdout:40: finishing write tid 3 to smithi19125844-40 2024-09-15T02:48:40.720 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 40 v 8 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-15T02:48:40.721 INFO:tasks.rados.rados.0.smithi191.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-15T02:48:40.721 INFO:tasks.rados.rados.0.smithi191.stdout:40: done (15 left) 2024-09-15T02:48:40.721 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 9 current snap is 0 2024-09-15T02:48:40.721 INFO:tasks.rados.rados.0.smithi191.stdout:64: seq_num 51 ranges {147456=491520,638976=376832} 2024-09-15T02:48:40.724 INFO:tasks.rados.rados.0.smithi191.stdout:64: writing smithi19125844-9 from 147456 to 638976 tid 1 2024-09-15T02:48:40.726 INFO:tasks.rados.rados.0.smithi191.stdout:64: writing smithi19125844-9 from 638976 to 1015808 tid 2 2024-09-15T02:48:40.726 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.733 INFO:tasks.rados.rados.0.smithi191.stdout:61: finishing copy_from racing read to smithi19125844-33 2024-09-15T02:48:40.735 INFO:tasks.rados.rados.0.smithi191.stdout:46: finishing write tid 3 to smithi19125844-46 2024-09-15T02:48:40.735 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-15T02:48:40.735 INFO:tasks.rados.rados.0.smithi191.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-15T02:48:40.735 INFO:tasks.rados.rados.0.smithi191.stdout:46: done (15 left) 2024-09-15T02:48:40.735 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 48 current snap is 0 2024-09-15T02:48:40.735 INFO:tasks.rados.rados.0.smithi191.stdout:65: seq_num 52 ranges {802816=507904,1310720=376832} 2024-09-15T02:48:40.736 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.736 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.744 INFO:tasks.rados.rados.0.smithi191.stdout:65: writing smithi19125844-48 from 802816 to 1310720 tid 1 2024-09-15T02:48:40.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:65: writing smithi19125844-48 from 1310720 to 1687552 tid 2 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:60: finishing copy_from racing read to smithi19125844-2 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:61: finishing copy_from to smithi19125844-33 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 33 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:55: finishing copy_from racing read to smithi19125844-41 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:61: done (15 left) 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 19 current snap is 0 2024-09-15T02:48:40.749 INFO:tasks.rados.rados.0.smithi191.stdout:66: seq_num 53 ranges {311296=524288,835584=376832} 2024-09-15T02:48:40.753 INFO:tasks.rados.rados.0.smithi191.stdout:66: writing smithi19125844-19 from 311296 to 835584 tid 1 2024-09-15T02:48:40.754 INFO:tasks.rados.rados.0.smithi191.stdout:66: writing smithi19125844-19 from 835584 to 1212416 tid 2 2024-09-15T02:48:40.755 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.756 INFO:tasks.rados.rados.0.smithi191.stdout:53: finishing copy_from racing read to smithi19125844-37 2024-09-15T02:48:40.756 INFO:tasks.rados.rados.0.smithi191.stdout:60: finishing copy_from to smithi19125844-2 2024-09-15T02:48:40.756 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 2 v 8 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T02:48:40.757 INFO:tasks.rados.rados.0.smithi191.stdout:60: done (15 left) 2024-09-15T02:48:40.757 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 8 current snap is 0 2024-09-15T02:48:40.757 INFO:tasks.rados.rados.0.smithi191.stdout:67: seq_num 54 ranges {131072=540672,671744=376832} 2024-09-15T02:48:40.761 INFO:tasks.rados.rados.0.smithi191.stdout:67: writing smithi19125844-8 from 131072 to 671744 tid 1 2024-09-15T02:48:40.764 INFO:tasks.rados.rados.0.smithi191.stdout:67: writing smithi19125844-8 from 671744 to 1048576 tid 2 2024-09-15T02:48:40.764 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.764 INFO:tasks.rados.rados.0.smithi191.stdout:53: finishing copy_from to smithi19125844-37 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 37 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:53: done (15 left) 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:68: read oid 25 snap -1 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:68: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 16 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:65: finishing write tid 1 to smithi19125844-48 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:65: finishing write tid 2 to smithi19125844-48 2024-09-15T02:48:40.765 INFO:tasks.rados.rados.0.smithi191.stdout:66: finishing write tid 1 to smithi19125844-19 2024-09-15T02:48:40.766 INFO:tasks.rados.rados.0.smithi191.stdout:65: finishing write tid 3 to smithi19125844-48 2024-09-15T02:48:40.768 INFO:tasks.rados.rados.0.smithi191.stdout:63: finishing write tid 1 to smithi19125844-5 2024-09-15T02:48:40.768 INFO:tasks.rados.rados.0.smithi191.stdout:63: finishing write tid 2 to smithi19125844-5 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:66: finishing write tid 2 to smithi19125844-19 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:65: finishing write tid 4 to smithi19125844-48 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 8 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:65: left oid 48 (ObjNum 52 snap 0 seq_num 52) 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:65: done (15 left) 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:69: delete oid 23 current snap is 0 2024-09-15T02:48:40.769 INFO:tasks.rados.rados.0.smithi191.stdout:66: finishing write tid 3 to smithi19125844-19 2024-09-15T02:48:40.770 INFO:tasks.rados.rados.0.smithi191.stdout:68: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-15T02:48:40.782 INFO:tasks.rados.rados.0.smithi191.stdout:66: finishing write tid 4 to smithi19125844-19 2024-09-15T02:48:40.782 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 19 v 9 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-15T02:48:40.782 INFO:tasks.rados.rados.0.smithi191.stdout:66: left oid 19 (ObjNum 53 snap 0 seq_num 53) 2024-09-15T02:48:40.782 INFO:tasks.rados.rados.0.smithi191.stdout:63: finishing write tid 3 to smithi19125844-5 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:64: finishing write tid 1 to smithi19125844-9 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:55: finishing copy_from to smithi19125844-41 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 41 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:64: finishing write tid 2 to smithi19125844-9 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:64: finishing write tid 3 to smithi19125844-9 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:55: done (15 left) 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:66: done (14 left) 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:68: done (13 left) 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:69: done (12 left) 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:70: rmattr oid 7 current snap is 0 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:70: done (12 left) 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:71: delete oid 3 current snap is 0 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:63: finishing write tid 4 to smithi19125844-5 2024-09-15T02:48:40.783 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 5 v 15 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-15T02:48:40.784 INFO:tasks.rados.rados.0.smithi191.stdout:63: left oid 5 (ObjNum 50 snap 0 seq_num 50) 2024-09-15T02:48:40.784 INFO:tasks.rados.rados.0.smithi191.stdout:67: finishing write tid 1 to smithi19125844-8 2024-09-15T02:48:40.784 INFO:tasks.rados.rados.0.smithi191.stdout:67: finishing write tid 2 to smithi19125844-8 2024-09-15T02:48:40.784 INFO:tasks.rados.rados.0.smithi191.stdout:64: finishing write tid 4 to smithi19125844-9 2024-09-15T02:48:40.784 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 9 v 19 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T02:48:40.784 INFO:tasks.rados.rados.0.smithi191.stdout:64: left oid 9 (ObjNum 51 snap 0 seq_num 51) 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:63: done (12 left) 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:64: done (11 left) 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:71: done (10 left) 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:72: copy_from oid 18 from oid 19 current snap is 0 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:73: rollback oid 44 current snap is 0 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:73: done (11 left) 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:74: read oid 6 snap -1 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:74: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T02:48:40.786 INFO:tasks.rados.rados.0.smithi191.stdout:75: delete oid 45 current snap is 0 2024-09-15T02:48:40.791 INFO:tasks.rados.rados.0.smithi191.stdout:75: done (12 left) 2024-09-15T02:48:40.792 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 13 current snap is 0 2024-09-15T02:48:40.792 INFO:tasks.rados.rados.0.smithi191.stdout:76: seq_num 58 ranges {212992=606208,819200=376832} 2024-09-15T02:48:40.801 INFO:tasks.rados.rados.0.smithi191.stdout:76: writing smithi19125844-13 from 212992 to 819200 tid 1 2024-09-15T02:48:40.803 INFO:tasks.rados.rados.0.smithi191.stdout:76: writing smithi19125844-13 from 819200 to 1196032 tid 2 2024-09-15T02:48:40.804 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 14 current snap is 0 2024-09-15T02:48:40.804 INFO:tasks.rados.rados.0.smithi191.stdout:74: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T02:48:40.806 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:40.801+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:40.810 INFO:tasks.rados.rados.0.smithi191.stdout:77: seq_num 59 ranges {229376=622592,851968=376832} 2024-09-15T02:48:40.816 INFO:tasks.rados.rados.0.smithi191.stdout:77: writing smithi19125844-14 from 229376 to 851968 tid 1 2024-09-15T02:48:40.819 INFO:tasks.rados.rados.0.smithi191.stdout:77: writing smithi19125844-14 from 851968 to 1228800 tid 2 2024-09-15T02:48:40.820 INFO:tasks.rados.rados.0.smithi191.stdout:74: done (13 left) 2024-09-15T02:48:40.820 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 37 current snap is 0 2024-09-15T02:48:40.820 INFO:tasks.rados.rados.0.smithi191.stdout:78: seq_num 60 ranges {65536=638976,704512=376832} 2024-09-15T02:48:40.825 INFO:tasks.rados.rados.0.smithi191.stdout:78: writing smithi19125844-37 from 65536 to 704512 tid 1 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:78: writing smithi19125844-37 from 704512 to 1081344 tid 2 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:79: rollback oid 48 current snap is 0 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:79: done (14 left) 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:80: snap_create 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:72: finishing copy_from racing read to smithi19125844-18 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:72: finishing copy_from to smithi19125844-18 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 18 v 9 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-15T02:48:40.828 INFO:tasks.rados.rados.0.smithi191.stdout:67: finishing write tid 3 to smithi19125844-8 2024-09-15T02:48:40.829 INFO:tasks.rados.rados.0.smithi191.stdout:34: finishing write tid 1 to smithi19125844-34 2024-09-15T02:48:40.829 INFO:tasks.rados.rados.0.smithi191.stdout:34: finishing write tid 2 to smithi19125844-34 2024-09-15T02:48:40.829 INFO:tasks.rados.rados.0.smithi191.stdout:42: finishing write tid 1 to smithi19125844-42 2024-09-15T02:48:40.831 INFO:tasks.rados.rados.0.smithi191.stdout:28: finishing write tid 1 to smithi19125844-28 2024-09-15T02:48:40.833 INFO:tasks.rados.rados.0.smithi191.stdout:34: finishing write tid 3 to smithi19125844-34 2024-09-15T02:48:40.833 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 34 v 2 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-15T02:48:40.833 INFO:tasks.rados.rados.0.smithi191.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-15T02:48:40.833 INFO:tasks.rados.rados.0.smithi191.stdout:76: finishing write tid 1 to smithi19125844-13 2024-09-15T02:48:40.835 INFO:tasks.rados.rados.0.smithi191.stdout:35: finishing write tid 1 to smithi19125844-35 2024-09-15T02:48:40.835 INFO:tasks.rados.rados.0.smithi191.stdout:77: finishing write tid 1 to smithi19125844-14 2024-09-15T02:48:40.835 INFO:tasks.rados.rados.0.smithi191.stdout:35: finishing write tid 2 to smithi19125844-35 2024-09-15T02:48:40.837 INFO:tasks.rados.rados.0.smithi191.stdout:77: finishing write tid 2 to smithi19125844-14 2024-09-15T02:48:40.837 INFO:tasks.rados.rados.0.smithi191.stdout:77: finishing write tid 3 to smithi19125844-14 2024-09-15T02:48:40.838 INFO:tasks.rados.rados.0.smithi191.stdout:35: finishing write tid 3 to smithi19125844-35 2024-09-15T02:48:40.838 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 35 v 2 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-15T02:48:40.838 INFO:tasks.rados.rados.0.smithi191.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-15T02:48:40.840 INFO:tasks.rados.rados.0.smithi191.stdout:42: finishing write tid 2 to smithi19125844-42 2024-09-15T02:48:40.841 INFO:tasks.rados.rados.0.smithi191.stdout:78: finishing write tid 1 to smithi19125844-37 2024-09-15T02:48:40.844 INFO:tasks.rados.rados.0.smithi191.stdout:78: finishing write tid 2 to smithi19125844-37 2024-09-15T02:48:40.844 INFO:tasks.rados.rados.0.smithi191.stdout:78: finishing write tid 3 to smithi19125844-37 2024-09-15T02:48:40.846 INFO:tasks.rados.rados.0.smithi191.stdout:77: finishing write tid 4 to smithi19125844-14 2024-09-15T02:48:40.846 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 12 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-15T02:48:40.846 INFO:tasks.rados.rados.0.smithi191.stdout:77: left oid 14 (ObjNum 59 snap 0 seq_num 59) 2024-09-15T02:48:40.847 INFO:tasks.rados.rados.0.smithi191.stdout:42: finishing write tid 3 to smithi19125844-42 2024-09-15T02:48:40.847 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 42 v 4 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T02:48:40.847 INFO:tasks.rados.rados.0.smithi191.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-15T02:48:40.847 INFO:tasks.rados.rados.0.smithi191.stdout:28: finishing write tid 2 to smithi19125844-28 2024-09-15T02:48:40.848 INFO:tasks.rados.rados.0.smithi191.stdout:47: finishing write tid 1 to smithi19125844-47 2024-09-15T02:48:40.848 INFO:tasks.rados.rados.0.smithi191.stdout:47: finishing write tid 2 to smithi19125844-47 2024-09-15T02:48:40.849 INFO:tasks.rados.rados.0.smithi191.stdout:78: finishing write tid 4 to smithi19125844-37 2024-09-15T02:48:40.849 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 37 v 13 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-15T02:48:40.849 INFO:tasks.rados.rados.0.smithi191.stdout:78: left oid 37 (ObjNum 60 snap 0 seq_num 60) 2024-09-15T02:48:40.849 INFO:tasks.rados.rados.0.smithi191.stdout:49: finishing write tid 1 to smithi19125844-49 2024-09-15T02:48:40.849 INFO:tasks.rados.rados.0.smithi191.stdout:49: finishing write tid 2 to smithi19125844-49 2024-09-15T02:48:40.886 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.886 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:40.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:48:40.906 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:40.901+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:40.908 INFO:teuthology.orchestra.run.smithi152.stdout:fsck success 2024-09-15T02:48:40.923 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-15T02:48:40.923 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-09-15T02:48:40.923 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-15T02:48:40.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-15T02:48:40.926 INFO:tasks.ceph.osd.10:Started 2024-09-15T02:48:40.926 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:48:40.994 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:40.994 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:48:41.007 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:41.005+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:41.029 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-15T02:48:41.029 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-15T02:48:41.029 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-15T02:48:41.029 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-15T02:48:41.029 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-15T02:48:41.029 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-15T02:48:41.030 INFO:teuthology.orchestra.run.smithi152.stderr:config diff 2024-09-15T02:48:41.030 INFO:teuthology.orchestra.run.smithi152.stderr:config diff get 2024-09-15T02:48:41.030 INFO:teuthology.orchestra.run.smithi152.stderr:config get 2024-09-15T02:48:41.030 INFO:teuthology.orchestra.run.smithi152.stderr:config help [] 2024-09-15T02:48:41.030 INFO:teuthology.orchestra.run.smithi152.stderr:config set ... 2024-09-15T02:48:41.030 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-15T02:48:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T02:48:41.032 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-15T02:48:41.091 INFO:tasks.rados.rados.0.smithi191.stdout:30: finishing write tid 2 to smithi19125844-30 2024-09-15T02:48:41.094 INFO:tasks.rados.rados.0.smithi191.stdout:30: finishing write tid 3 to smithi19125844-30 2024-09-15T02:48:41.094 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 30 v 8 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-15T02:48:41.094 INFO:tasks.rados.rados.0.smithi191.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-15T02:48:41.109 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:41.103+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:48:41.210 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:41.203+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:41.238 INFO:tasks.rados.rados.0.smithi191.stdout:28: finishing write tid 3 to smithi19125844-28 2024-09-15T02:48:41.239 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 2 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-15T02:48:41.239 INFO:tasks.rados.rados.0.smithi191.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-15T02:48:41.242 INFO:tasks.rados.rados.0.smithi191.stdout:47: finishing write tid 3 to smithi19125844-47 2024-09-15T02:48:41.242 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 47 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-15T02:48:41.242 INFO:tasks.rados.rados.0.smithi191.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-15T02:48:41.262 INFO:tasks.rados.rados.0.smithi191.stdout:50: finishing write tid 3 to smithi19125844-50 2024-09-15T02:48:41.265 INFO:tasks.rados.rados.0.smithi191.stdout:50: finishing write tid 4 to smithi19125844-50 2024-09-15T02:48:41.265 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 50 v 5 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-15T02:48:41.266 INFO:tasks.rados.rados.0.smithi191.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-15T02:48:41.269 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.269 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:48:41.311 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:41.305+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:41.342 INFO:tasks.rados.rados.0.smithi191.stdout:49: finishing write tid 3 to smithi19125844-49 2024-09-15T02:48:41.379 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.379 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:48:41.412 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:41.409+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:41.483 INFO:tasks.rados.rados.0.smithi191.stdout:76: finishing write tid 2 to smithi19125844-13 2024-09-15T02:48:41.490 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.490 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:48:41.525 INFO:tasks.rados.rados.0.smithi191.stdout:28: done (14 left) 2024-09-15T02:48:41.525 INFO:tasks.rados.rados.0.smithi191.stdout:30: done (13 left) 2024-09-15T02:48:41.525 INFO:tasks.rados.rados.0.smithi191.stdout:34: done (12 left) 2024-09-15T02:48:41.525 INFO:tasks.rados.rados.0.smithi191.stdout:35: done (11 left) 2024-09-15T02:48:41.525 INFO:tasks.rados.rados.0.smithi191.stdout:42: done (10 left) 2024-09-15T02:48:41.525 INFO:tasks.rados.rados.0.smithi191.stdout:47: done (9 left) 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:50: done (8 left) 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:72: done (7 left) 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:77: done (6 left) 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:78: done (5 left) 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:80: done (4 left) 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 24 current snap is 1 2024-09-15T02:48:41.526 INFO:tasks.rados.rados.0.smithi191.stdout:81: seq_num 61 ranges {393216=655360,1048576=376832} 2024-09-15T02:48:41.530 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:48:41.539 INFO:tasks.rados.rados.0.smithi191.stdout:81: writing smithi19125844-24 from 393216 to 1048576 tid 1 2024-09-15T02:48:41.541 INFO:tasks.rados.rados.0.smithi191.stdout:81: writing smithi19125844-24 from 1048576 to 1425408 tid 2 2024-09-15T02:48:41.541 INFO:tasks.rados.rados.0.smithi191.stdout:82: snap_remove snap 0 2024-09-15T02:48:41.548 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:41.545+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:41.585 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:41.582+0000 7f8942297540 -1 Falling back to public interface 2024-09-15T02:48:41.614 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.614 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.614 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:41.609+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:41.623 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:48:41.715 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:41.709+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:41.721 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.721 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:48:41.816 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:41.807+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:41.827 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.828 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.836 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:48:41.917 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:41.914+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:41.939 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:41.939 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:41.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:48:42.018 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:42.014+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:42.048 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.048 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.057 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:48:42.118 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:42.114+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:42.155 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.156 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:48:42.219 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:42.213+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:42.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:48:42.265 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.265 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:48:42.320 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:42.317+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:42.371 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.371 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:48:42.389 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:42.417 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:42.421 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:42.417+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:42.446 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:42.475 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:42.483 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.483 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.492 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:48:42.503 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:42.522 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:42.517+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:42.534 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:42.535 INFO:tasks.rados.rados.0.smithi191.stdout:82: done (5 left) 2024-09-15T02:48:42.535 INFO:tasks.rados.rados.0.smithi191.stdout:83: snap_create 2024-09-15T02:48:42.535 INFO:tasks.rados.rados.0.smithi191.stdout:67: finishing write tid 4 to smithi19125844-8 2024-09-15T02:48:42.535 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 8 v 22 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-09-15T02:48:42.535 INFO:tasks.rados.rados.0.smithi191.stdout:67: left oid 8 (ObjNum 54 snap 0 seq_num 54) 2024-09-15T02:48:42.535 INFO:tasks.rados.rados.0.smithi191.stdout:57: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T02:48:42.546 INFO:tasks.rados.rados.0.smithi191.stdout:76: finishing write tid 3 to smithi19125844-13 2024-09-15T02:48:42.546 INFO:tasks.rados.rados.0.smithi191.stdout:81: finishing write tid 1 to smithi19125844-24 2024-09-15T02:48:42.546 INFO:tasks.rados.rados.0.smithi191.stdout:81: finishing write tid 2 to smithi19125844-24 2024-09-15T02:48:42.546 INFO:tasks.rados.rados.0.smithi191.stdout:49: finishing write tid 4 to smithi19125844-49 2024-09-15T02:48:42.546 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 49 v 7 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:76: finishing write tid 4 to smithi19125844-13 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 13 v 8 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:76: left oid 13 (ObjNum 58 snap 0 seq_num 58) 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:81: finishing write tid 3 to smithi19125844-24 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:81: finishing write tid 4 to smithi19125844-24 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 24 v 12 (ObjNum 61 snap 1 seq_num 61) dirty exists 2024-09-15T02:48:42.547 INFO:tasks.rados.rados.0.smithi191.stdout:81: left oid 24 (ObjNum 61 snap 1 seq_num 61) 2024-09-15T02:48:42.594 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.595 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:48:42.622 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:42.615+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:42.661 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:42.693 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:42.700 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.700 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:48:42.722 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:42.723 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:42.718+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:42.748 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:42.778 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:42.806 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:42.808 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:42.808 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:42.817 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:48:42.824 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:42.815+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:42.832 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:42.859 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:42.925 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:42.922+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:42.926 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:42.926 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:42.926 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:42.926 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:42.936 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:48:43.025 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:43.022+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:43.042 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:43.043 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.052 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:48:43.126 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:43.125+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:43.157 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:20 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'10, mlcod=24'8)", 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.452670+0000", 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.700511448, 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052649309999999996, 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.158 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452670+0000", 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452670+0000", 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452672+0000", 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.906e-06 2024-09-15T02:48:43.159 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452868+0000", 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019606300000000001 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452955+0000", 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6203000000000001e-05 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.160 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452966+0000", 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1521e-05 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.453092+0000", 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012625099999999999 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.453158+0000", 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5332000000000004e-05 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.161 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.457827+0000", 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0046690009999999999 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.457935+0000", 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010865399999999999 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:22 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'12, mlcod=24'8)", 2024-09-15T02:48:43.162 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.455497+0000", 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.697684841000001, 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0049479609999999999, 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.455497+0000", 2024-09-15T02:48:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.455497+0000", 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.455499+0000", 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.164 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.455654+0000", 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000155372 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.455722+0000", 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8004999999999997e-05 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.455731+0000", 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0939999999999997e-06 2024-09-15T02:48:43.165 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.456434+0000", 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00070336699999999999 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.456486+0000", 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1581e-05 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.460366+0000", 2024-09-15T02:48:43.166 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0038797789999999999 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.460445+0000", 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9163000000000001e-05 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:41 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'23, mlcod=24'19)", 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.729948+0000", 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.423233404000001, 2024-09-15T02:48:43.167 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052060489999999999, 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.729948+0000", 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.729948+0000", 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.168 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.729949+0000", 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1999999999999999e-07 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730305+0000", 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00035559099999999999 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730470+0000", 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000165012 2024-09-15T02:48:43.169 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730504+0000", 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4325000000000001e-05 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730828+0000", 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032381700000000001 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730856+0000", 2024-09-15T02:48:43.170 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7864999999999998e-05 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735100+0000", 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042445809999999999 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735155+0000", 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4138000000000002e-05 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.171 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'19)", 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730319+0000", 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.422863306, 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0049214710000000002, 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730319+0000", 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.172 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730319+0000", 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730319+0000", 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9400000000000001e-07 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730323+0000", 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9899999999999999e-06 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730565+0000", 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024201800000000001 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730570+0000", 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7929999999999997e-06 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.731760+0000", 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011898270000000001 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.731792+0000", 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2904000000000002e-05 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735202+0000", 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0034099199999999999 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.175 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735240+0000", 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7625e-05 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:44 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'26, mlcod=24'19)", 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730354+0000", 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.422828000999999, 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004936691, 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.176 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730354+0000", 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730354+0000", 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730354+0000", 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:48:43.177 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730405+0000", 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1202000000000002e-05 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730583+0000", 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017749899999999999 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730588+0000", 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6770000000000003e-06 2024-09-15T02:48:43.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732363+0000", 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001775499 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732398+0000", 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4523999999999997e-05 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735248+0000", 2024-09-15T02:48:43.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028507879999999999 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735291+0000", 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2162000000000001e-05 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:45 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'27, mlcod=24'19)", 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730409+0000", 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.422772893000001, 2024-09-15T02:48:43.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0049126600000000001, 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730409+0000", 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730409+0000", 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730409+0000", 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4999999999999998e-07 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730413+0000", 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1370000000000004e-06 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730600+0000", 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018605600000000001 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730604+0000", 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4270000000000001e-06 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732948+0000", 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002344311 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733019+0000", 2024-09-15T02:48:43.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0796000000000005e-05 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735297+0000", 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022779860000000001 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735322+0000", 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4596999999999998e-05 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:46 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'28, mlcod=24'19)", 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.422764863000001, 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070834360000000002, 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5999999999999999e-07 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730421+0000", 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8770000000000003e-06 2024-09-15T02:48:43.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730887+0000", 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000465253 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730893+0000", 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7100000000000001e-06 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733567+0000", 2024-09-15T02:48:43.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002673922 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733619+0000", 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1465000000000003e-05 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.737397+0000", 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0037785319999999998 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.737500+0000", 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000103317 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:47 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'29, mlcod=24'19)", 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730800+0000", 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.422382066000001, 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0072831750000000002, 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.189 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730800+0000", 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730800+0000", 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730801+0000", 2024-09-15T02:48:43.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.176e-06 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730850+0000", 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8541999999999997e-05 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732102+0000", 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012526810000000001 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732118+0000", 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6198e-05 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734170+0000", 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0020519150000000001 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734198+0000", 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7614e-05 2024-09-15T02:48:43.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738021+0000", 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0038232800000000001 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738083+0000", 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.1768999999999998e-05 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:48 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'30, mlcod=24'20)", 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.732043+0000", 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.421138761999998, 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060923130000000002, 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732043+0000", 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.194 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732043+0000", 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732045+0000", 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6199999999999999e-06 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732060+0000", 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5387999999999999e-05 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732930+0000", 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00086952500000000001 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732943+0000", 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2947e-05 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734641+0000", 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016985640000000001 2024-09-15T02:48:43.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734674+0000", 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2977e-05 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738088+0000", 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0034136269999999998 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738135+0000", 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7664999999999998e-05 2024-09-15T02:48:43.197 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:49 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'31, mlcod=24'20)", 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.732838+0000", 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.420343962, 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053466640000000001, 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.198 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732838+0000", 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732838+0000", 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732840+0000", 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732879+0000", 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9944999999999998e-05 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733806+0000", 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00092643399999999996 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733840+0000", 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4187999999999999e-05 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735397+0000", 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0015569780000000001 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735431+0000", 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4381000000000002e-05 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738139+0000", 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0027073539999999999 2024-09-15T02:48:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738185+0000", 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.416901117999998, 2024-09-15T02:48:43.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:48:43.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:48:43.206 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:71 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'49, mlcod=24'45)", 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.339997+0000", 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.813184890999999, 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0065061889999999999, 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.207 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.339997+0000", 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.339997+0000", 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.339999+0000", 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6199999999999999e-06 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340011+0000", 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2801e-05 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340091+0000", 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0065e-05 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340102+0000", 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0484e-05 2024-09-15T02:48:43.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340192+0000", 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0513000000000006e-05 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340240+0000", 2024-09-15T02:48:43.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7845e-05 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346425+0000", 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0061847940000000004 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346503+0000", 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8066999999999996e-05 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:72 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'50, mlcod=24'46)", 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.340749+0000", 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.812432707999999, 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058094349999999999, 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340749+0000", 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340749+0000", 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340751+0000", 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.347e-06 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340808+0000", 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7008e-05 2024-09-15T02:48:43.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340873+0000", 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5611999999999995e-05 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340885+0000", 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.15e-05 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340964+0000", 2024-09-15T02:48:43.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9312000000000002e-05 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.341002+0000", 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8331999999999998e-05 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346514+0000", 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055117480000000003 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346559+0000", 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4576000000000001e-05 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:73 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'51, mlcod=24'46)", 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.342123+0000", 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.811058394, 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070136319999999997, 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.216 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342123+0000", 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342123+0000", 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.217 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342125+0000", 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.254e-06 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342137+0000", 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2027e-05 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342206+0000", 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9605999999999997e-05 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.218 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342216+0000", 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3899999999999999e-06 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345367+0000", 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0031511669999999999 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345413+0000", 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6516e-05 2024-09-15T02:48:43.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349036+0000", 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036222989999999998 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349137+0000", 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010137300000000001 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:74 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'52, mlcod=24'47)", 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.342890+0000", 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.810291463999999, 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0063358019999999998, 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342890+0000", 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342890+0000", 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342891+0000", 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0999999999999998e-07 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342899+0000", 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9580000000000007e-06 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.222 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342967+0000", 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8088000000000003e-05 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343024+0000", 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6577999999999999e-05 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345902+0000", 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028785220000000001 2024-09-15T02:48:43.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345946+0000", 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3603999999999998e-05 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349145+0000", 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003198986 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349226+0000", 2024-09-15T02:48:43.224 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1355999999999998e-05 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'47)", 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.343614+0000", 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.809568351999999, 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0059812909999999997, 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343614+0000", 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343614+0000", 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343615+0000", 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.257e-06 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343689+0000", 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4176000000000003e-05 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343772+0000", 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3279e-05 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343781+0000", 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0559999999999994e-06 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.227 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346598+0000", 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028166129999999999 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346631+0000", 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3498999999999999e-05 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349465+0000", 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002833534 2024-09-15T02:48:43.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349595+0000", 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000129877 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:99 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'73, mlcod=25'66)", 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.579821+0000", 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.573360456, 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055442140000000004, 2024-09-15T02:48:43.229 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579821+0000", 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579821+0000", 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579823+0000", 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0899999999999999e-06 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579938+0000", 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000115767 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579991+0000", 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2797999999999998e-05 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.580001+0000", 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0003999999999999e-05 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.580473+0000", 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00047201200000000001 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.580489+0000", 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6399e-05 2024-09-15T02:48:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.585327+0000", 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048373829999999998 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.585366+0000", 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8760999999999997e-05 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:112 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'82, mlcod=25'80)", 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:26.377777+0000", 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.775405030000002, 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050900160000000002, 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377777+0000", 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.234 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377777+0000", 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377778+0000", 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.576e-06 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377791+0000", 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2078e-05 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377874+0000", 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3535000000000006e-05 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377885+0000", 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1311000000000001e-05 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.377984+0000", 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8739000000000004e-05 2024-09-15T02:48:43.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.378036+0000", 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2234999999999997e-05 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.382799+0000", 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0047626400000000003 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:26.382867+0000", 2024-09-15T02:48:43.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7902000000000001e-05 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:137 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'103, mlcod=25'100)", 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:28.314615+0000", 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.838567075, 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053880100000000004, 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.238 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314615+0000", 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314615+0000", 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314616+0000", 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.305e-06 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314630+0000", 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3436e-05 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314695+0000", 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5398e-05 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314703+0000", 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3760000000000008e-06 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314809+0000", 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010540900000000001 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314854+0000", 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5571999999999997e-05 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.319943+0000", 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050882210000000004 2024-09-15T02:48:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.320003+0000", 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0293e-05 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:138 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'104, mlcod=25'100)", 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:28.315552+0000", 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.837629982999999, 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.005117702, 2024-09-15T02:48:43.242 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315552+0000", 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315552+0000", 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315553+0000", 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.13e-06 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315742+0000", 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018857300000000001 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315795+0000", 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3579000000000003e-05 2024-09-15T02:48:43.244 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315806+0000", 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0808000000000001e-05 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315902+0000", 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5922999999999996e-05 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.315958+0000", 2024-09-15T02:48:43.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6066999999999997e-05 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.320627+0000", 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0046692469999999996 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.320670+0000", 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2375e-05 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.246 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.247 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.247 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.248 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:48:43.249 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:48:43.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:48:43.251 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:43.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:48:43.251 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:43.241+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:43.349 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:43.343+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:43.355 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.355 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:43.355 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:43.355 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.364 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:48:43.450 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:43.446+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:43.462 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.462 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:43.462 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:43.462 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:43.462 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.471 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:48:43.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:49: done (5 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:57: done (4 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:67: done (3 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:76: done (2 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:81: done (1 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:83: done (0 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:84: rmattr oid 5 current snap is 2 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:84: done (0 left) 2024-09-15T02:48:43.551 INFO:tasks.rados.rados.0.smithi191.stdout:85: setattr oid 45 current snap is 2 2024-09-15T02:48:43.557 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 1 current snap is 2 2024-09-15T02:48:43.557 INFO:tasks.rados.rados.0.smithi191.stdout:86: seq_num 62 ranges {32768=671744,704512=376832} 2024-09-15T02:48:43.563 INFO:tasks.rados.rados.0.smithi191.stdout:86: writing smithi19125844-1 from 32768 to 704512 tid 1 2024-09-15T02:48:43.566 INFO:tasks.rados.rados.0.smithi191.stdout:86: writing smithi19125844-1 from 704512 to 1081344 tid 2 2024-09-15T02:48:43.566 INFO:tasks.rados.rados.0.smithi191.stdout:87: read oid 10 snap -1 2024-09-15T02:48:43.566 INFO:tasks.rados.rados.0.smithi191.stdout:87: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-15T02:48:43.566 INFO:tasks.rados.rados.0.smithi191.stdout:88: rollback oid 46 current snap is 2 2024-09-15T02:48:43.566 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 46 to 1 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 45 v 14 (ObjNum 225 snap 0 seq_num 564680400) dirty exists 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:85: done (3 left) 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:89: rollback oid 36 current snap is 2 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 36 to 1 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:90: rollback oid 27 current snap is 2 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 27 to 1 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:91: read oid 20 snap -1 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:91: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T02:48:43.567 INFO:tasks.rados.rados.0.smithi191.stdout:92: snap_create 2024-09-15T02:48:43.574 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.574 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:43.575 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:43.575 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [] 2024-09-15T02:48:43.575 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.577 INFO:tasks.rados.rados.0.smithi191.stdout:87: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-15T02:48:43.585 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:48:43.585 INFO:tasks.rados.rados.0.smithi191.stdout:90: finishing rollback tid 0 to smithi19125844-27 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:89: finishing rollback tid 0 to smithi19125844-36 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:88: finishing rollback tid 0 to smithi19125844-46 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:89: finishing rollback tid 1 to smithi19125844-36 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:90: finishing rollback tid 1 to smithi19125844-27 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:88: finishing rollback tid 1 to smithi19125844-46 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:88: finishing rollback tid 2 to smithi19125844-46 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 46 v 26 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:90: finishing rollback tid 2 to smithi19125844-27 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 27 v 18 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T02:48:43.586 INFO:tasks.rados.rados.0.smithi191.stdout:91: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T02:48:43.588 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:43.583+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:43.594 INFO:tasks.rados.rados.0.smithi191.stdout:86: finishing write tid 1 to smithi19125844-1 2024-09-15T02:48:43.602 INFO:tasks.rados.rados.0.smithi191.stdout:86: finishing write tid 2 to smithi19125844-1 2024-09-15T02:48:43.602 INFO:tasks.rados.rados.0.smithi191.stdout:86: finishing write tid 3 to smithi19125844-1 2024-09-15T02:48:43.604 INFO:tasks.rados.rados.0.smithi191.stdout:86: finishing write tid 4 to smithi19125844-1 2024-09-15T02:48:43.605 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 1 v 12 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-09-15T02:48:43.605 INFO:tasks.rados.rados.0.smithi191.stdout:86: left oid 1 (ObjNum 62 snap 2 seq_num 62) 2024-09-15T02:48:43.651 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:43.649+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:43.681 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.681 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:43.681 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:43.681 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.690 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:48:43.752 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:43.750+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:43.785 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.785 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:43.785 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:43.785 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:43.786 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.794 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:48:43.853 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:43.849+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:237 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.32537159199999999, 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013334751000000001, 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.895 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 237 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565426+0000", 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2820000000000001e-06 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565442+0000", 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6096999999999999e-05 2024-09-15T02:48:43.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565484+0000", 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2092999999999998e-05 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565500+0000", 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5976000000000001e-05 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565647+0000", 2024-09-15T02:48:43.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014730900000000001 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565664+0000", 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6405e-05 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569523+0000", 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003859627 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569647+0000", 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000123292 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578757+0000", 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0091106699999999995 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:238 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:43.900 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.32527125600000001, 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016905343999999999, 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 238 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:43.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565524+0000", 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2619999999999999e-06 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565531+0000", 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3300000000000001e-06 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565555+0000", 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3272999999999999e-05 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565567+0000", 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.205e-05 2024-09-15T02:48:43.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565812+0000", 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024487000000000001 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565822+0000", 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0356999999999999e-05 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.904 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050794789999999996 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571110+0000", 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020819100000000001 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582428+0000", 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011318531999999999 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.905 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:239 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.32519710600000001, 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017215766, 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:43.906 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 239 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565598+0000", 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0669999999999999e-06 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565604+0000", 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7069999999999997e-06 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565633+0000", 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9546000000000001e-05 2024-09-15T02:48:43.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565642+0000", 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0950000000000002e-06 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565870+0000", 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00022745799999999999 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565885+0000", 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.525e-05 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571578+0000", 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056929210000000001 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571650+0000", 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1894e-05 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582813+0000", 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011162828 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.9s0 39/33 ECSubRead(tid=25, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.570163+0000", 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.32063075200000002, 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063556699999999997, 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.911 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570163+0000", 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570163+0000", 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570163+0000", 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570163+0000", 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570184+0000", 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1138999999999999e-05 2024-09-15T02:48:43.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570195+0000", 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0533000000000001e-05 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570269+0000", 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3589000000000003e-05 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570799+0000", 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00053030600000000003 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.320007667, 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001307866, 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.916 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570852+0000", 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5881000000000002e-05 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570868+0000", 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5452e-05 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572031+0000", 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011636719999999999 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572094+0000", 2024-09-15T02:48:43.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2860999999999993e-05 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.9s0 39/33 ECSubRead(tid=26, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.319325374, 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00082345300000000003, 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571509+0000", 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0441999999999997e-05 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571520+0000", 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0655e-05 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572172+0000", 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065217599999999997 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572292+0000", 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012018 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.923 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.9s0 39/33 ECSubRead(tid=27, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31887541600000002, 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00070841600000000002, 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.925 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572343+0000", 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042437699999999998 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572351+0000", 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3049999999999993e-06 2024-09-15T02:48:43.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572525+0000", 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017411299999999999 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572627+0000", 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010162099999999999 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.927 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.572137+0000", 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31865696500000001, 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00036803200000000002, 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.928 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572137+0000", 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572137+0000", 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572138+0000", 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.085e-06 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572230+0000", 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1705000000000004e-05 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572282+0000", 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2014e-05 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572292+0000", 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0343000000000001e-05 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572462+0000", 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000170151 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572505+0000", 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2734000000000001e-05 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.572287+0000", 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31850719900000002, 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00041398500000000001, 2024-09-15T02:48:43.932 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572287+0000", 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572287+0000", 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572287+0000", 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572287+0000", 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572377+0000", 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0433999999999995e-05 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572380+0000", 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0869999999999998e-06 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572652+0000", 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027190999999999999 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572701+0000", 2024-09-15T02:48:43.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8554000000000003e-05 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.572606+0000", 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31818780299999999, 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000393309, 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572606+0000", 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572606+0000", 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572607+0000", 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8899999999999998e-07 2024-09-15T02:48:43.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572700+0000", 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3004999999999995e-05 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572885+0000", 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000184511 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572893+0000", 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8469999999999996e-06 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572949+0000", 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6297e-05 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.573000+0000", 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0659999999999999e-05 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.939 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.572620+0000", 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31817421000000001, 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015628800000000001, 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572620+0000", 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572620+0000", 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572620+0000", 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572620+0000", 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572638+0000", 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8352e-05 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572645+0000", 2024-09-15T02:48:43.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.4579999999999998e-06 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572728+0000", 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3765999999999997e-05 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572776+0000", 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7712e-05 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.572712+0000", 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31808206, 2024-09-15T02:48:43.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00040286400000000001, 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572712+0000", 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572712+0000", 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572712+0000", 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4000000000000002e-07 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572843+0000", 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000130579 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.946 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572914+0000", 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1260000000000006e-05 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572919+0000", 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6149999999999999e-06 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.573068+0000", 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000148925 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.573115+0000", 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7045000000000001e-05 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.576678+0000", 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31411588899999998, 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000322243, 2024-09-15T02:48:43.948 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576678+0000", 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576678+0000", 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576679+0000", 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2419999999999999e-06 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576865+0000", 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018604299999999999 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576908+0000", 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.299e-05 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576916+0000", 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0360000000000006e-06 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576961+0000", 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4762000000000003e-05 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577000+0000", 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9169999999999999e-05 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.577206+0000", 2024-09-15T02:48:43.952 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31358809399999998, 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031080099999999998, 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577206+0000", 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577206+0000", 2024-09-15T02:48:43.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577207+0000", 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7899999999999998e-07 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577307+0000", 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000100263 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577421+0000", 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000113732 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577426+0000", 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9629999999999997e-06 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577474+0000", 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8386000000000001e-05 2024-09-15T02:48:43.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577517+0000", 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2778000000000003e-05 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.577310+0000", 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.313483605, 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00025748999999999998, 2024-09-15T02:48:43.956 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577310+0000", 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577310+0000", 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577311+0000", 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9200000000000002e-07 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577387+0000", 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6463000000000003e-05 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577454+0000", 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6342999999999998e-05 2024-09-15T02:48:43.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577458+0000", 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2100000000000003e-06 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577535+0000", 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6808000000000001e-05 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.959 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577568+0000", 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3374000000000003e-05 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.578065+0000", 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.31272940799999999, 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071418700000000002, 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578065+0000", 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578065+0000", 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578066+0000", 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4419999999999999e-06 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578232+0000", 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016620099999999999 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578274+0000", 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2132999999999998e-05 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578281+0000", 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5200000000000003e-06 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578333+0000", 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1780999999999998e-05 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578779+0000", 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00044611 2024-09-15T02:48:43.963 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.581704+0000", 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.30909012200000002, 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00075500399999999998, 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.964 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581704+0000", 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581704+0000", 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581705+0000", 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.02e-06 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581782+0000", 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7280000000000005e-05 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581905+0000", 2024-09-15T02:48:43.968 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000123103 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581913+0000", 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0010000000000001e-06 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.969 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581993+0000", 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9396999999999999e-05 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582459+0000", 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000466203 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-15T02:48:43.970 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.30896734599999998, 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00099855899999999999, 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.971 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3399999999999999e-07 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581992+0000", 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016515799999999999 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.972 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582026+0000", 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3154999999999997e-05 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582031+0000", 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4569999999999996e-06 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582494+0000", 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00046331600000000002 2024-09-15T02:48:43.973 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582825+0000", 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00033083899999999998 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:43.974 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:43.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:48:43.977 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:43.969+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:44.057 INFO:tasks.rados.rados.0.smithi191.stdout:89: finishing rollback tid 2 to smithi19125844-36 2024-09-15T02:48:44.057 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 36 v 11 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-15T02:48:44.077 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:44.069+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:44.086 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:44.086 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:48:44.178 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:44.174+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:44.195 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:44.195 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:48:44.251 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:44.246+0000 7f8942297540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:48:44.279 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:44.273+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:44.304 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.73371448399999994, 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:48:44.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.72377217800000004, 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099046280000000004, 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579386+0000", 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.536e-06 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579616+0000", 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230366 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579660+0000", 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4489e-05 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579668+0000", 2024-09-15T02:48:44.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4800000000000004e-06 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579731+0000", 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2733999999999999e-05 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579742+0000", 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589278+0000", 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095358219999999994 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589289+0000", 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0596e-05 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582826+0000", 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.72033025399999995, 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064948000000000002, 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582826+0000", 2024-09-15T02:48:44.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582826+0000", 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582827+0000", 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5300000000000003e-07 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582848+0000", 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0890000000000002e-05 2024-09-15T02:48:44.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582897+0000", 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9184999999999997e-05 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582906+0000", 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2080000000000006e-06 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583564+0000", 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065821499999999997 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583574+0000", 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9820000000000003e-06 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589310+0000", 2024-09-15T02:48:44.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057356109999999998 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589321+0000", 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0856000000000001e-05 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=97, reqid=client.4526.0:223, at_version=0'0, trim_to=0'0, roll_forward_to=39'12))", 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.602279+0000", 2024-09-15T02:48:44.320 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.700877109, 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011563400000000001, 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602279+0000", 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602279+0000", 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602281+0000", 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.502e-06 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602301+0000", 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0216000000000001e-05 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602372+0000", 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1353999999999995e-05 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602381+0000", 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2930000000000006e-06 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602491+0000", 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000109768 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602513+0000", 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.158e-05 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603347+0000", 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083392399999999999 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603435+0000", 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8702999999999994e-05 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.1s2 39/33 ECSubRead(tid=98, to_read={3:8610d5f9:::smithi19125844-1:head=0,4096,0}, subchunks={3:8610d5f9:::smithi19125844-1:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.602902+0000", 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.70025400299999996, 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028495899999999999, 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.325 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602902+0000", 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602902+0000", 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602903+0000", 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.02e-06 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602918+0000", 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.451e-05 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602970+0000", 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2024999999999997e-05 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602982+0000", 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2459e-05 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603057+0000", 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4828999999999998e-05 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603187+0000", 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130116 2024-09-15T02:48:44.328 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.329 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.329 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.329 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.329 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.329 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:48:44.380 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:44.377+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:44.476 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.477 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:44.477 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:44.477 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.481 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:44.475+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:44.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:48:44.555 INFO:tasks.rados.rados.0.smithi191.stdout:86: done (6 left) 2024-09-15T02:48:44.555 INFO:tasks.rados.rados.0.smithi191.stdout:87: done (5 left) 2024-09-15T02:48:44.555 INFO:tasks.rados.rados.0.smithi191.stdout:88: done (4 left) 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:89: done (3 left) 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:90: done (2 left) 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:91: done (1 left) 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:92: done (0 left) 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:93: read oid 47 snap -1 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:93: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 8 current snap is 3 2024-09-15T02:48:44.556 INFO:tasks.rados.rados.0.smithi191.stdout:94: seq_num 63 ranges {1048576=688128,1736704=376832} 2024-09-15T02:48:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:48:44.568 INFO:tasks.rados.rados.0.smithi191.stdout:94: writing smithi19125844-8 from 1048576 to 1736704 tid 1 2024-09-15T02:48:44.570 INFO:tasks.rados.rados.0.smithi191.stdout:94: writing smithi19125844-8 from 1736704 to 2113536 tid 2 2024-09-15T02:48:44.570 INFO:tasks.rados.rados.0.smithi191.stdout:95: snap_remove snap 2 2024-09-15T02:48:44.582 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:44.575+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:44.591 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.591 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:44.591 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:44.592 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:44.592 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:48:44.683 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:44.678+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:44.697 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 39/33 ECSubWrite(tid=91, reqid=client.4526.0:228, at_version=39'24, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.1262664929999999, 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048861590000000002, 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9400000000000005e-07 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569139+0000", 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7068999999999999e-05 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569277+0000", 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013777200000000001 2024-09-15T02:48:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569299+0000", 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2033000000000001e-05 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569495+0000", 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000195336 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569523+0000", 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.879e-05 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.573920+0000", 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043971050000000001 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.573968+0000", 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7360000000000001e-05 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.1256071910000001, 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:48:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 39/33 ECSubWrite(tid=92, reqid=client.4526.0:229, at_version=39'25, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.578450+0000", 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.1168978540000001, 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002119087, 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578450+0000", 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578450+0000", 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578451+0000", 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.181e-06 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578477+0000", 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5174e-05 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578527+0000", 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0754000000000002e-05 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578538+0000", 2024-09-15T02:48:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0419e-05 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578987+0000", 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044919799999999999 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578996+0000", 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2960000000000004e-06 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580531+0000", 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015345199999999999 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580569+0000", 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8544999999999998e-05 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.1130910059999999, 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:48:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 39/33 ECSubWrite(tid=94, reqid=client.4526.0:230, at_version=39'26, trim_to=0'0, roll_forward_to=39'24))", 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582397+0000", 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.112951443, 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00079575800000000003, 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582397+0000", 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582397+0000", 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582397+0000", 2024-09-15T02:48:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9200000000000002e-07 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582406+0000", 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3109999999999995e-06 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582483+0000", 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6420000000000004e-05 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582487+0000", 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1300000000000003e-06 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582535+0000", 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8402000000000002e-05 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582545+0000", 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4800000000000007e-06 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583165+0000", 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061998800000000003 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583192+0000", 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7634999999999999e-05 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.112051846, 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:48:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:48:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 39/33 ECSubWrite(tid=96, reqid=client.4526.0:230, at_version=0'0, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.584166+0000", 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.111182203, 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001130612, 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584166+0000", 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584166+0000", 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584167+0000", 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2900000000000002e-07 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584178+0000", 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1661000000000001e-05 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584224+0000", 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5668999999999999e-05 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584256+0000", 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2030000000000003e-05 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584313+0000", 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7018000000000002e-05 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584325+0000", 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2343e-05 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.585258+0000", 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093266200000000001 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.585297+0000", 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8399999999999998e-05 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=97, reqid=client.4526.0:223, at_version=0'0, trim_to=0'0, roll_forward_to=39'12))", 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.602274+0000", 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.0930738950000001, 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087540199999999995, 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602274+0000", 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602274+0000", 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602275+0000", 2024-09-15T02:48:44.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1699999999999997e-07 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602289+0000", 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4212999999999999e-05 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602337+0000", 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8084999999999998e-05 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602345+0000", 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7069999999999992e-06 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602410+0000", 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5094999999999999e-05 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602423+0000", 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2830000000000001e-05 2024-09-15T02:48:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603107+0000", 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068364399999999996 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603150+0000", 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2911000000000003e-05 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as2 40/33 ECSubRead(tid=18, to_read={3:517ce0e6:::smithi19125844-47:head=0,196608,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.556587+0000", 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.13876134200000001, 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010856220000000001, 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556587+0000", 2024-09-15T02:48:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556587+0000", 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556588+0000", 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6000000000000003e-07 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556604+0000", 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6480000000000001e-05 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556719+0000", 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011459 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556743+0000", 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4229000000000001e-05 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556904+0000", 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016105500000000001 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.557672+0000", 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00076850799999999999 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as2 40/33 ECSubRead(tid=19, to_read={3:517ce0e6:::smithi19125844-47:head=0,196608,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.558332+0000", 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.13701601799999999, 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028025899999999999, 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558332+0000", 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558332+0000", 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558334+0000", 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5099999999999999e-06 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558366+0000", 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2756999999999998e-05 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558406+0000", 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9496000000000003e-05 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558424+0000", 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8476999999999999e-05 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558496+0000", 2024-09-15T02:48:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1526999999999995e-05 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558612+0000", 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116492 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as2 40/33 ECSubRead(tid=20, to_read={3:517ce0e6:::smithi19125844-47:head=0,196608,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.559542+0000", 2024-09-15T02:48:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.13580615200000001, 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035264900000000002, 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559542+0000", 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559542+0000", 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559543+0000", 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3430000000000001e-06 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559562+0000", 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8777e-05 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559614+0000", 2024-09-15T02:48:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2061999999999997e-05 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559635+0000", 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0432e-05 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559721+0000", 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6743000000000006e-05 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.559895+0000", 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017329199999999999 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.11763217300000001, 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081904660000000004, 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4799999999999999e-07 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578238+0000", 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052198700000000002 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578284+0000", 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5556000000000002e-05 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2568000000000001e-05 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578387+0000", 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0867999999999998e-05 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578406+0000", 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8995999999999999e-05 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.585856+0000", 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074492930000000001 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.585906+0000", 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0850000000000003e-05 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 40/33 ECSubWrite(tid=100, reqid=client.4526.0:245, at_version=40'29, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.589294+0000", 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.106053674, 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047200760000000001, 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589294+0000", 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589294+0000", 2024-09-15T02:48:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589296+0000", 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1599999999999999e-06 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589437+0000", 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014099300000000001 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589488+0000", 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0937999999999999e-05 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589499+0000", 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1022000000000001e-05 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589569+0000", 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0623000000000005e-05 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589582+0000", 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2649999999999999e-05 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.593939+0000", 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043576379999999996 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.594015+0000", 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5051999999999999e-05 2024-09-15T02:48:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 40/33 ECSubWrite(tid=101, reqid=client.4526.0:246, at_version=40'30, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.592907+0000", 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.10244136199999999, 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00161443, 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.592907+0000", 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.592907+0000", 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.592908+0000", 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1659999999999999e-06 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.592937+0000", 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8904e-05 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.593000+0000", 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3588000000000002e-05 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.593012+0000", 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1728e-05 2024-09-15T02:48:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.593533+0000", 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052086099999999996 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.593554+0000", 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1214999999999999e-05 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.594477+0000", 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00092232600000000001 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.594521+0000", 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4641999999999997e-05 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 40/33 ECSubWrite(tid=102, reqid=client.4526.0:246, at_version=0'0, trim_to=0'0, roll_forward_to=40'30))", 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.600988+0000", 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.094360419000000001, 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00097243099999999995, 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.600988+0000", 2024-09-15T02:48:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.600988+0000", 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.600990+0000", 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1519999999999999e-06 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601014+0000", 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4363e-05 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601083+0000", 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8308999999999999e-05 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601095+0000", 2024-09-15T02:48:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2842e-05 2024-09-15T02:48:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601222+0000", 2024-09-15T02:48:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012644899999999999 2024-09-15T02:48:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601237+0000", 2024-09-15T02:48:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5682999999999999e-05 2024-09-15T02:48:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601912+0000", 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067435599999999998 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601960+0000", 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8276999999999999e-05 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.7s2 40/33 ECSubRead(tid=103, to_read={3:e10d9d4d:::smithi19125844-8:head=0,4096,0}, subchunks={3:e10d9d4d:::smithi19125844-8:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.601737+0000", 2024-09-15T02:48:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.093611076000000001, 2024-09-15T02:48:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033182399999999999, 2024-09-15T02:48:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601737+0000", 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601737+0000", 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601739+0000", 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6139999999999999e-06 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601760+0000", 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1719999999999999e-05 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601891+0000", 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130691 2024-09-15T02:48:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601911+0000", 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9748000000000001e-05 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.601988+0000", 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7279000000000003e-05 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.602069+0000", 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0772000000000005e-05 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:44.775 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:48:44.786 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:44.781+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:44.879 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.880 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:44.880 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:44.880 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.887 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:44.881+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:44.889 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:48:44.988 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:44.988 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:44.988 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:44.988 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:44.989 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:44.989 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:44.985+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:44.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:48:45.054 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:45.050+0000 7f8942297540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:48:45.089 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:45.086+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:45.098 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:45.098 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:45.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:45.098 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:45.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.098 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.729432794000001, 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:48:45.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:48:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:48:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723076+0000", 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.370102600999999, 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016520386000000001, 2024-09-15T02:48:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723076+0000", 2024-09-15T02:48:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723076+0000", 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723077+0000", 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.009e-06 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723081+0000", 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5000000000000001e-06 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723106+0000", 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4300000000000001e-05 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723121+0000", 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5617e-05 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.732427+0000", 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093053460000000008 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.732582+0000", 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015555699999999999 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.732799+0000", 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021657299999999999 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738098+0000", 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052989100000000004 2024-09-15T02:48:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739475+0000", 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013773609999999999 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739546+0000", 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1459999999999997e-05 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739596+0000", 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9753000000000003e-05 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723130+0000", 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.370048788999998, 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016698528000000001, 2024-09-15T02:48:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 49 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723130+0000", 2024-09-15T02:48:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723130+0000", 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723131+0000", 2024-09-15T02:48:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.569999999999999e-07 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723137+0000", 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3230000000000003e-06 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723145+0000", 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.303e-06 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723150+0000", 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9139999999999999e-06 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733435+0000", 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010284887 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733500+0000", 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5455000000000003e-05 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733600+0000", 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9709000000000004e-05 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738112+0000", 2024-09-15T02:48:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045116660000000001 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739689+0000", 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015773779999999999 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739778+0000", 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8541000000000005e-05 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739828+0000", 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0395000000000001e-05 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.370016918000001, 2024-09-15T02:48:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016944363, 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 50 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723162+0000", 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5799999999999999e-07 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723225+0000", 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3168000000000002e-05 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723248+0000", 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3031999999999999e-05 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723255+0000", 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8889999999999996e-06 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.733941+0000", 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010686161 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.734161+0000", 2024-09-15T02:48:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219189 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.734379+0000", 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021872800000000001 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738124+0000", 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003744561 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.739946+0000", 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001821629 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740056+0000", 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011092800000000001 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740106+0000", 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9419999999999998e-05 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723270+0000", 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.369908869, 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017134052, 2024-09-15T02:48:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 51 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723270+0000", 2024-09-15T02:48:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723270+0000", 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723271+0000", 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.172e-06 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723278+0000", 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.746e-06 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723297+0000", 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8513999999999999e-05 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723307+0000", 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6369999999999994e-06 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.735565+0000", 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012258118 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.735689+0000", 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012389500000000001 2024-09-15T02:48:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.735896+0000", 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000207772 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.738136+0000", 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002239343 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740202+0000", 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020667289999999998 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740288+0000", 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740404+0000", 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116017 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.369861279999999, 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:48:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:48:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:48:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.761271285999999, 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018893783000000001, 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331908+0000", 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.133e-06 2024-09-15T02:48:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331914+0000", 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5799999999999999e-06 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331925+0000", 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0893999999999999e-05 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331931+0000", 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2199999999999997e-06 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342666+0000", 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01073519 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342753+0000", 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7175999999999993e-05 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342888+0000", 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013462700000000001 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348168+0000", 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005280285 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350557+0000", 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023890309999999998 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350698+0000", 2024-09-15T02:48:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140755 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350801+0000", 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102892 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.761240816000001, 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026305674000000001, 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331943+0000", 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7559999999999999e-06 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331955+0000", 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-09-15T02:48:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331964+0000", 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4069999999999993e-06 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343483+0000", 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011518995000000001 2024-09-15T02:48:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343581+0000", 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8568999999999998e-05 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343713+0000", 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013195700000000001 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348183+0000", 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044698639999999996 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358029+0000", 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0098460219999999994 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358143+0000", 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114335 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358243+0000", 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9881000000000001e-05 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.761208139000001, 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026584729000000001, 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331971+0000", 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332135+0000", 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000164098 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332158+0000", 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332165+0000", 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8669999999999996e-06 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344120+0000", 2024-09-15T02:48:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011955222 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344205+0000", 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4895000000000006e-05 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344336+0000", 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130844 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.349645+0000", 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053089770000000003 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358356+0000", 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087112079999999998 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358462+0000", 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105664 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358555+0000", 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3110999999999997e-05 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.760998453999999, 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:48:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:48:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:48:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.524526613999999, 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017901001, 2024-09-15T02:48:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 98 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568653+0000", 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4700000000000001e-07 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568658+0000", 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.006e-06 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568666+0000", 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5369999999999997e-06 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568671+0000", 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3700000000000003e-06 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580006+0000", 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011334946 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580064+0000", 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7207999999999998e-05 2024-09-15T02:48:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580154+0000", 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0682999999999999e-05 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.582087+0000", 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019328520000000001 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586462+0000", 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004374452 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586503+0000", 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1872000000000002e-05 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586553+0000", 2024-09-15T02:48:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9327999999999999e-05 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.524424509999999, 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017954067000000001, 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 99 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568755+0000", 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9300000000000006e-07 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568847+0000", 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1906999999999999e-05 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568859+0000", 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2047e-05 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568866+0000", 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7170000000000002e-06 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580440+0000", 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011574442000000001 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580500+0000", 2024-09-15T02:48:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9732999999999999e-05 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580611+0000", 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011137999999999999 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583173+0000", 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025617330000000001 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586614+0000", 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034406929999999999 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586671+0000", 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7398000000000002e-05 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586708+0000", 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7024000000000003e-05 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.524304835999999, 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:48:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:48:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:48:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:48:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:48:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:48:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.506786259999998, 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:48:45.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:48:45.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:48:45.201 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.692923833000002, 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:48:45.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:48:45.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:48:45.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:48:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:48:45.209 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313294+0000", 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.779884427999999, 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016588219000000001, 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.210 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313294+0000", 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313294+0000", 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313296+0000", 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9130000000000001e-06 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313485+0000", 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018899699999999999 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313507+0000", 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.226e-05 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313520+0000", 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3027000000000001e-05 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.315971+0000", 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024512280000000002 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316058+0000", 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6513000000000003e-05 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316201+0000", 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014322200000000001 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320589+0000", 2024-09-15T02:48:45.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043881939999999998 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329755+0000", 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091656010000000007 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329815+0000", 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9849999999999998e-05 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329882+0000", 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7414000000000002e-05 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313537+0000", 2024-09-15T02:48:45.217 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.779641616999999, 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016521168999999999, 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313537+0000", 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313537+0000", 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313539+0000", 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8700000000000001e-06 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313548+0000", 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4860000000000003e-06 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313592+0000", 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3717000000000002e-05 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313622+0000", 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.968e-05 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316761+0000", 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003139537 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.316875+0000", 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011346200000000001 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.317026+0000", 2024-09-15T02:48:45.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015164099999999999 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320605+0000", 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003578332 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.329945+0000", 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093409329999999992 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330016+0000", 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0519999999999996e-05 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330058+0000", 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1990999999999998e-05 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313636+0000", 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.779541977000001, 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016775641000000001, 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.225 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313636+0000", 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313636+0000", 2024-09-15T02:48:45.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313638+0000", 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.57e-06 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313647+0000", 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2529999999999993e-06 2024-09-15T02:48:45.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313665+0000", 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8003e-05 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313691+0000", 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5264000000000001e-05 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.317796+0000", 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041053410000000002 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.317908+0000", 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111722 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.318100+0000", 2024-09-15T02:48:45.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019205099999999999 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320618+0000", 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002518831 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330209+0000", 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095900470000000008 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330342+0000", 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013296599999999999 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330412+0000", 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0593000000000007e-05 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313720+0000", 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.779458587000001, 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016893389000000002, 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.232 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313720+0000", 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313720+0000", 2024-09-15T02:48:45.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313721+0000", 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.406e-06 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313870+0000", 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014921800000000001 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313897+0000", 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6812999999999999e-05 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313948+0000", 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000001e-05 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.318791+0000", 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048430310000000002 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.318931+0000", 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000139603 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.319168+0000", 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023724600000000001 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.321596+0000", 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024279729999999999 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330505+0000", 2024-09-15T02:48:45.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0089086970000000001 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330558+0000", 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3513000000000001e-05 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330613+0000", 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5179000000000001e-05 2024-09-15T02:48:45.238 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:142 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:28.313967+0000", 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.779211356000001, 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017001057999999999, 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.239 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 142 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313967+0000", 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313967+0000", 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313969+0000", 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.55e-06 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313979+0000", 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0603e-05 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.313997+0000", 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7629999999999999e-05 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.314012+0000", 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4756999999999999e-05 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320653+0000", 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066416599999999998 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320783+0000", 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012967299999999999 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:48:45.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.320967+0000", 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000183821 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.322180+0000", 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012135830000000001 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330877+0000", 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0086965759999999993 2024-09-15T02:48:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330925+0000", 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8513999999999997e-05 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.330968+0000", 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2691000000000002e-05 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.245 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.246 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.246 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:45.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:48:45.248 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:45.239+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:45.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:48:45.349 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:45.345+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:45.355 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:45.355 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:45.355 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:45.355 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:45.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:48:45.376 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:45.374+0000 7f8942297540 -1 osd.10 29 log_to_monitors true 2024-09-15T02:48:45.408 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:45.438 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:45.450 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:45.446+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:45.458 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:45.458 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:45.458 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:45.458 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:45.459 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:45.466 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:45.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:48:45.493 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:45.519 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:45.547 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:45.552 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:45.545+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:45.567 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/33 ECSubWriteReply(tid=19, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.0230695609999998, 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74030938599999996, 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:48:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539611+0000", 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3680000000000001e-06 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539619+0000", 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278674+0000", 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73905563200000002 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278687+0000", 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.239e-05 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278784+0000", 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7149000000000001e-05 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.279919+0000", 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001135645 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/33 ECSubWriteReply(tid=20, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.281193+0000", 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2814868239999999, 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028968499999999998, 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281193+0000", 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281193+0000", 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281194+0000", 2024-09-15T02:48:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.72e-06 2024-09-15T02:48:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281205+0000", 2024-09-15T02:48:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0030000000000001e-05 2024-09-15T02:48:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281281+0000", 2024-09-15T02:48:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6419000000000001e-05 2024-09-15T02:48:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281298+0000", 2024-09-15T02:48:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6609999999999999e-05 2024-09-15T02:48:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281453+0000", 2024-09-15T02:48:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015515900000000001 2024-09-15T02:48:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.281482+0000", 2024-09-15T02:48:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9747000000000002e-05 2024-09-15T02:48:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:234 3.3s0 3:cebed992:::smithi19125844-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.565967+0000", 2024-09-15T02:48:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9967126689999999, 2024-09-15T02:48:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012231558, 2024-09-15T02:48:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:48:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 234 2024-09-15T02:48:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565967+0000", 2024-09-15T02:48:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565967+0000", 2024-09-15T02:48:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565969+0000", 2024-09-15T02:48:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.891e-06 2024-09-15T02:48:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565976+0000", 2024-09-15T02:48:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0570000000000004e-06 2024-09-15T02:48:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565997+0000", 2024-09-15T02:48:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0678000000000001e-05 2024-09-15T02:48:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566005+0000", 2024-09-15T02:48:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2349999999999999e-06 2024-09-15T02:48:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566086+0000", 2024-09-15T02:48:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1392999999999998e-05 2024-09-15T02:48:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:48:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566098+0000", 2024-09-15T02:48:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1361e-05 2024-09-15T02:48:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570199+0000", 2024-09-15T02:48:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041015019999999999 2024-09-15T02:48:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570317+0000", 2024-09-15T02:48:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117787 2024-09-15T02:48:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571897+0000", 2024-09-15T02:48:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001580674 2024-09-15T02:48:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577044+0000", 2024-09-15T02:48:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051465269999999997 2024-09-15T02:48:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578083+0000", 2024-09-15T02:48:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001038612 2024-09-15T02:48:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578198+0000", 2024-09-15T02:48:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115841 2024-09-15T02:48:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:235 3.3s0 3:cebed992:::smithi19125844-27:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.566032+0000", 2024-09-15T02:48:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9966472319999999, 2024-09-15T02:48:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014205251, 2024-09-15T02:48:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:48:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 235 2024-09-15T02:48:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566032+0000", 2024-09-15T02:48:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566032+0000", 2024-09-15T02:48:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566033+0000", 2024-09-15T02:48:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6300000000000005e-07 2024-09-15T02:48:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566048+0000", 2024-09-15T02:48:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5051999999999999e-05 2024-09-15T02:48:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566056+0000", 2024-09-15T02:48:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:48:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566066+0000", 2024-09-15T02:48:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8630000000000003e-06 2024-09-15T02:48:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566224+0000", 2024-09-15T02:48:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000157637 2024-09-15T02:48:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566250+0000", 2024-09-15T02:48:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6570000000000001e-05 2024-09-15T02:48:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572420+0000", 2024-09-15T02:48:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006169557 2024-09-15T02:48:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572522+0000", 2024-09-15T02:48:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101848 2024-09-15T02:48:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.573712+0000", 2024-09-15T02:48:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011904319999999999 2024-09-15T02:48:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577493+0000", 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037809860000000001 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580068+0000", 2024-09-15T02:48:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025749179999999998 2024-09-15T02:48:45.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580238+0000", 2024-09-15T02:48:45.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016964899999999999 2024-09-15T02:48:45.615 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.616 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.616 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.616 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:236 3.3s0 3:cebed992:::smithi19125844-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:45.616 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:48:45.617 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9966070490000001, 2024-09-15T02:48:45.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018295445, 2024-09-15T02:48:45.617 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.617 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:45.617 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:48:45.618 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:48:45.618 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:45.618 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 236 2024-09-15T02:48:45.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.618 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:48:45.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1600000000000001e-07 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566075+0000", 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2359999999999999e-06 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566083+0000", 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1410000000000005e-06 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566091+0000", 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8529999999999992e-06 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566291+0000", 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000199242 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566295+0000", 2024-09-15T02:48:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.972e-06 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574064+0000", 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077684240000000003 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574167+0000", 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010275 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:45.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574540+0000", 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037325999999999999 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577496+0000", 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002956242 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584322+0000", 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0068258470000000003 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584368+0000", 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5961999999999998e-05 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.9s3 39/33 ECSubRead(tid=25, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.571000+0000", 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9916791819999999, 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014937800000000001, 2024-09-15T02:48:45.624 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571000+0000", 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571000+0000", 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571001+0000", 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.34e-07 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571010+0000", 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0089999999999996e-06 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571066+0000", 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6424000000000001e-05 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571074+0000", 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.157e-06 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571474+0000", 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039922999999999998 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572494+0000", 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001020626 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.9s3 39/33 ECSubRead(tid=26, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.572248+0000", 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9904313810000001, 2024-09-15T02:48:45.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000668449, 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572248+0000", 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572248+0000", 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572250+0000", 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8899999999999999e-06 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572271+0000", 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1137000000000001e-05 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572405+0000", 2024-09-15T02:48:45.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013395599999999999 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572425+0000", 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9493000000000001e-05 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572566+0000", 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014088600000000001 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572917+0000", 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035108699999999998 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=21, last_complete=39'16, committed=1, applied=1))", 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.574119+0000", 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.988560689, 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069745400000000002, 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.632 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574119+0000", 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574119+0000", 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574120+0000", 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1299999999999998e-07 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574127+0000", 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8179999999999998e-06 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574167+0000", 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0352000000000003e-05 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574174+0000", 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9550000000000003e-06 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574798+0000", 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00062393300000000002 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574816+0000", 2024-09-15T02:48:45.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8482999999999999e-05 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=21, last_complete=39'16, committed=1, applied=1))", 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.574593+0000", 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.988086961, 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030164999999999999, 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574593+0000", 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574593+0000", 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574594+0000", 2024-09-15T02:48:45.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1399999999999995e-07 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574599+0000", 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5690000000000004e-06 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574633+0000", 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3454000000000002e-05 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574639+0000", 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.754e-06 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574877+0000", 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023806200000000001 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574894+0000", 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6897000000000001e-05 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=22, last_complete=39'17, committed=1, applied=1))", 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.574667+0000", 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.98801212, 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029487900000000001, 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.640 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574667+0000", 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574667+0000", 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574668+0000", 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9200000000000001e-07 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574671+0000", 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574709+0000", 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7542000000000001e-05 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574725+0000", 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6113999999999999e-05 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574931+0000", 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020596300000000001 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574962+0000", 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1361e-05 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.643 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=21, last_complete=39'16, committed=1, applied=1))", 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577604+0000", 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9850759629999999, 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061371699999999995, 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577604+0000", 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577604+0000", 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577605+0000", 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.035e-06 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577612+0000", 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9020000000000002e-06 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577665+0000", 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3804000000000002e-05 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577673+0000", 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3089999999999998e-06 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577867+0000", 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019469600000000001 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578217+0000", 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034997099999999997 2024-09-15T02:48:45.647 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=22, last_complete=39'17, committed=1, applied=1))", 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577617+0000", 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9850629639999999, 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000651551, 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.648 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577617+0000", 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577617+0000", 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577617+0000", 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7300000000000002e-07 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577619+0000", 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5339999999999998e-06 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577697+0000", 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7534999999999995e-05 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577707+0000", 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6190000000000006e-06 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578249+0000", 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054292100000000003 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578268+0000", 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8669e-05 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=23, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577621+0000", 2024-09-15T02:48:45.652 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9850581809999999, 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068447799999999997, 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577621+0000", 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577621+0000", 2024-09-15T02:48:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577622+0000", 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0699999999999999e-07 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577624+0000", 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3609999999999999e-06 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577718+0000", 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4221999999999993e-05 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577723+0000", 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4649999999999996e-06 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578295+0000", 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057198700000000004 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578306+0000", 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1236000000000001e-05 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=23, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577635+0000", 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9850443530000002, 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074546499999999995, 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.656 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577635+0000", 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577635+0000", 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577636+0000", 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0589999999999999e-06 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577642+0000", 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2029999999999997e-06 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577728+0000", 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5100999999999997e-05 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577760+0000", 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2314000000000003e-05 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578369+0000", 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00060914599999999997 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578381+0000", 2024-09-15T02:48:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1642e-05 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=22, last_complete=39'17, committed=1, applied=1))", 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577654+0000", 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.985025319, 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00081565299999999995, 2024-09-15T02:48:45.660 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577654+0000", 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577654+0000", 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577655+0000", 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9900000000000002e-07 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577657+0000", 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5150000000000001e-06 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578360+0000", 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00070297000000000003 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578367+0000", 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3189999999999997e-06 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578459+0000", 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1737000000000006e-05 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578470+0000", 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0813e-05 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.664 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=23, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577663+0000", 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.985016576, 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084102000000000005, 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577663+0000", 2024-09-15T02:48:45.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577663+0000", 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577663+0000", 2024-09-15T02:48:45.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5499999999999999e-07 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577666+0000", 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2479999999999999e-06 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578380+0000", 2024-09-15T02:48:45.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071425099999999995 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578384+0000", 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3839999999999999e-06 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578494+0000", 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110216 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578504+0000", 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6660000000000003e-06 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=22, last_complete=39'17, committed=1, applied=1))", 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.577765+0000", 2024-09-15T02:48:45.669 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9849141210000001, 2024-09-15T02:48:45.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065793399999999999, 2024-09-15T02:48:45.670 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.670 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.670 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577765+0000", 2024-09-15T02:48:45.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577765+0000", 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577766+0000", 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.06e-07 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577772+0000", 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4779999999999999e-06 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577816+0000", 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4703999999999998e-05 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577823+0000", 2024-09-15T02:48:45.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5069999999999997e-06 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578405+0000", 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00058242600000000004 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578423+0000", 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8113000000000001e-05 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=22, last_complete=39'17, committed=1, applied=1))", 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579839+0000", 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.982840851, 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000416005, 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579839+0000", 2024-09-15T02:48:45.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579839+0000", 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579840+0000", 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0499999999999999e-06 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579846+0000", 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.105e-06 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579900+0000", 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3647000000000003e-05 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579907+0000", 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-09-15T02:48:45.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579965+0000", 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8696000000000002e-05 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580255+0000", 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028926899999999998 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=23, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.584088+0000", 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.978591875, 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077290499999999995, 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584088+0000", 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584088+0000", 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584089+0000", 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1340000000000001e-06 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584096+0000", 2024-09-15T02:48:45.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2599999999999999e-06 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584139+0000", 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3223999999999998e-05 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584163+0000", 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4107000000000001e-05 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584231+0000", 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7247000000000002e-05 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584861+0000", 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00062993299999999995 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.682 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=24, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9757022719999999, 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30338358999999998, 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586978+0000", 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.51e-07 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586981+0000", 2024-09-15T02:48:45.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890125+0000", 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30314395700000002 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890138+0000", 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2627e-05 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890297+0000", 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015958899999999999 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890361+0000", 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3658999999999995e-05 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.686 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:45.687 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:45.687 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:45.688 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:45.578+0000 7f89387de700 -1 osd.10 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:48:45.689 INFO:tasks.rados.rados.0.smithi191.stdout:95: done (2 left) 2024-09-15T02:48:45.690 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 29 current snap is 3 2024-09-15T02:48:45.690 INFO:tasks.rados.rados.0.smithi191.stdout:96: seq_num 64 ranges {475136=704512,1179648=376832} 2024-09-15T02:48:45.690 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:45.690 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:45.690 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:45.690 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:45.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:48:45.692 INFO:tasks.rados.rados.0.smithi191.stdout:96: writing smithi19125844-29 from 475136 to 1179648 tid 1 2024-09-15T02:48:45.692 INFO:tasks.rados.rados.0.smithi191.stdout:96: writing smithi19125844-29 from 1179648 to 1556480 tid 2 2024-09-15T02:48:45.692 INFO:tasks.rados.rados.0.smithi191.stdout:97: snap_create 2024-09-15T02:48:45.692 INFO:tasks.rados.rados.0.smithi191.stdout:93: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-15T02:48:45.692 INFO:tasks.rados.rados.0.smithi191.stdout:94: finishing write tid 1 to smithi19125844-8 2024-09-15T02:48:45.693 INFO:tasks.rados.rados.0.smithi191.stdout:94: finishing write tid 2 to smithi19125844-8 2024-09-15T02:48:45.693 INFO:tasks.rados.rados.0.smithi191.stdout:94: finishing write tid 3 to smithi19125844-8 2024-09-15T02:48:45.693 INFO:tasks.rados.rados.0.smithi191.stdout:94: finishing write tid 4 to smithi19125844-8 2024-09-15T02:48:45.693 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 8 v 30 (ObjNum 63 snap 3 seq_num 63) dirty exists 2024-09-15T02:48:45.693 INFO:tasks.rados.rados.0.smithi191.stdout:94: left oid 8 (ObjNum 63 snap 3 seq_num 63) 2024-09-15T02:48:45.693 INFO:tasks.rados.rados.0.smithi191.stdout:96: finishing write tid 1 to smithi19125844-29 2024-09-15T02:48:45.694 INFO:tasks.rados.rados.0.smithi191.stdout:96: finishing write tid 2 to smithi19125844-29 2024-09-15T02:48:45.694 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:45.681+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:45.694 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:45.720 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:45.747 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:45.777 INFO:tasks.rados.rados.0.smithi191.stdout:96: finishing write tid 3 to smithi19125844-29 2024-09-15T02:48:45.777 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:45.780 INFO:tasks.rados.rados.0.smithi191.stdout:96: finishing write tid 4 to smithi19125844-29 2024-09-15T02:48:45.780 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 29 v 15 (ObjNum 64 snap 3 seq_num 64) dirty exists 2024-09-15T02:48:45.780 INFO:tasks.rados.rados.0.smithi191.stdout:96: left oid 29 (ObjNum 64 snap 3 seq_num 64) 2024-09-15T02:48:45.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:45.779+0000 7f03277fe700 0 -- 172.21.15.6:0/739462691 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7f0328195b80 msgr2=0x7f0328197f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:48:45.789 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:45.785+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:45.811 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:45.812 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:45.813 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:45.813 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:48:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:48:45.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:48:45.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:48:45.891 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:45.886+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:45.920 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:45.920 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:45.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:48:45.992 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:45.985+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:46.033 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.033 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.033 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:48:46.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:48:46.093 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:46.089+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:46.140 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.141 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.143 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:46.143 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:46.143 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:46.143 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:46.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:48:46.153 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-15T02:48:46.193 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:46.187+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:46.254 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.254 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:48:46.295 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:46.290+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:46.395 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:46.390+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:46.406 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.406 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:48:46.496 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:46.487+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:46.521 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.521 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:48:46.578 INFO:tasks.rados.rados.0.smithi191.stdout:93: done (3 left) 2024-09-15T02:48:46.578 INFO:tasks.rados.rados.0.smithi191.stdout:94: done (2 left) 2024-09-15T02:48:46.578 INFO:tasks.rados.rados.0.smithi191.stdout:96: done (1 left) 2024-09-15T02:48:46.578 INFO:tasks.rados.rados.0.smithi191.stdout:97: done (0 left) 2024-09-15T02:48:46.579 INFO:tasks.rados.rados.0.smithi191.stdout:98: copy_from oid 29 from oid 9 current snap is 4 2024-09-15T02:48:46.579 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 48 current snap is 4 2024-09-15T02:48:46.579 INFO:tasks.rados.rados.0.smithi191.stdout:99: seq_num 65 ranges {1687552=720896,2408448=376832} 2024-09-15T02:48:46.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:48:46.597 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:46.594+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:46.599 INFO:tasks.rados.rados.0.smithi191.stdout:99: writing smithi19125844-48 from 1687552 to 2408448 tid 1 2024-09-15T02:48:46.602 INFO:tasks.rados.rados.0.smithi191.stdout:99: writing smithi19125844-48 from 2408448 to 2785280 tid 2 2024-09-15T02:48:46.602 INFO:tasks.rados.rados.0.smithi191.stdout:100: rollback oid 46 current snap is 4 2024-09-15T02:48:46.602 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 46 to 1 2024-09-15T02:48:46.602 INFO:tasks.rados.rados.0.smithi191.stdout:101: rollback oid 14 current snap is 4 2024-09-15T02:48:46.602 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 14 to 1 2024-09-15T02:48:46.603 INFO:tasks.rados.rados.0.smithi191.stdout:102: setattr oid 26 current snap is 4 2024-09-15T02:48:46.603 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 47 current snap is 4 2024-09-15T02:48:46.603 INFO:tasks.rados.rados.0.smithi191.stdout:103: seq_num 66 ranges {786432=737280,1523712=376832} 2024-09-15T02:48:46.612 INFO:tasks.rados.rados.0.smithi191.stdout:103: writing smithi19125844-47 from 786432 to 1523712 tid 1 2024-09-15T02:48:46.615 INFO:tasks.rados.rados.0.smithi191.stdout:103: writing smithi19125844-47 from 1523712 to 1900544 tid 2 2024-09-15T02:48:46.615 INFO:tasks.rados.rados.0.smithi191.stdout:104: copy_from oid 40 from oid 23 current snap is 4 2024-09-15T02:48:46.615 INFO:tasks.rados.rados.0.smithi191.stdout:100: finishing rollback tid 0 to smithi19125844-46 2024-09-15T02:48:46.615 INFO:tasks.rados.rados.0.smithi191.stdout:105: snap_remove snap 3 2024-09-15T02:48:46.698 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:46.693+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:46.706 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.706 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:48:46.799 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:46.793+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:46.813 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.813 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:48:46.900 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:46.895+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:46.917 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:46.917 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:46.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:48:47.001 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:46.997+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:47.022 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.023 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:48:47.103 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:47.098+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:47.132 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.132 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.141 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:48:47.204 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:47.197+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:47.250 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.250 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:48:47.305 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:47.301+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:47.365 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.365 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:48:47.406 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:47.401+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:47.473 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.473 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:48:47.506 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:47.501+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:47.586 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.587 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.594 INFO:tasks.rados.rados.0.smithi191.stdout:105: done (7 left) 2024-09-15T02:48:47.594 INFO:tasks.rados.rados.0.smithi191.stdout:106: delete oid 16 current snap is 4 2024-09-15T02:48:47.594 INFO:tasks.rados.rados.0.smithi191.stdout:104: finishing copy_from to smithi19125844-40 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:104: got expected ENOENT (src dne) 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:104: finishing copy_from racing read to smithi19125844-40 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:101: finishing rollback tid 0 to smithi19125844-14 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:101: finishing rollback tid 1 to smithi19125844-14 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:101: finishing rollback tid 2 to smithi19125844-14 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 16 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 26 v 10 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:100: finishing rollback tid 1 to smithi19125844-46 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:99: finishing write tid 1 to smithi19125844-48 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:99: finishing write tid 2 to smithi19125844-48 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:98: finishing copy_from racing read to smithi19125844-29 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:98: finishing copy_from to smithi19125844-29 2024-09-15T02:48:47.595 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 29 v 19 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:103: finishing write tid 1 to smithi19125844-47 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:103: finishing write tid 2 to smithi19125844-47 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:103: finishing write tid 3 to smithi19125844-47 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:103: finishing write tid 4 to smithi19125844-47 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 47 v 11 (ObjNum 66 snap 4 seq_num 66) dirty exists 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:103: left oid 47 (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:99: finishing write tid 3 to smithi19125844-48 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:99: finishing write tid 4 to smithi19125844-48 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 12 (ObjNum 65 snap 4 seq_num 65) dirty exists 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:99: left oid 48 (ObjNum 65 snap 4 seq_num 65) 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:100: finishing rollback tid 2 to smithi19125844-46 2024-09-15T02:48:47.596 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 46 v 36 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-15T02:48:47.598 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:48:47.601 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:48:47.608 INFO:tasks.rados.rados.0.smithi191.stdout:98: done (7 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:99: done (6 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:100: done (5 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:101: done (4 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:102: done (3 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:103: done (2 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:104: done (1 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:106: done (0 left) 2024-09-15T02:48:47.609 INFO:tasks.rados.rados.0.smithi191.stdout:107: setattr oid 48 current snap is 4 2024-09-15T02:48:47.610 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:47.603+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:47.610 INFO:tasks.rados.rados.0.smithi191.stdout:108: copy_from oid 14 from oid 10 current snap is 4 2024-09-15T02:48:47.610 INFO:tasks.rados.rados.0.smithi191.stdout:109: copy_from oid 20 from oid 27 current snap is 4 2024-09-15T02:48:47.611 INFO:tasks.rados.rados.0.smithi191.stdout:110: snap_remove snap 1 2024-09-15T02:48:47.709 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:47.703+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:47.741 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.741 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:48:47.809 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:47.803+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:47.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:48:47.855 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.855 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:48:47.910 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:47.905+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:47.961 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:47.961 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:47.970 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:48:47.976 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:48.007 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:48.011 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:48.003+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:48.036 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:48.066 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:48.070 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:48.070 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:48.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:48:48.094 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:48.119 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:48.124 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:48.121+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:48.148 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:48.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:48.180 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:48.180 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:48.189 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:48:48.211 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:48.214 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:48.209+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:48.239 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:48.268 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:48:48.286 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:48.286 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:48.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:48:48.297 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:48.315 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:48.309+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:48.324 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:48.352 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:48.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:48.371+0000 7f741bfff700 0 -- 172.21.15.6:0/388654265 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f741c1955d0 msgr2=0x7f741c1979b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:48:48.388 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:48.395 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:48:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:48:48.395 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:48.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:48:48.416 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:48.414+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:48.419 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:48.517 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:48.509+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:48.547 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:48.547 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:221 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~671744] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.563671+0000", 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9818954209999999, 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031205257, 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.548 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.563671+0000", 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.563671+0000", 2024-09-15T02:48:48.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.563674+0000", 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2499999999999998e-06 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565217+0000", 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015434089999999999 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565261+0000", 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4045999999999998e-05 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565278+0000", 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6883000000000001e-05 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565423+0000", 2024-09-15T02:48:48.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000144776 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565489+0000", 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6361999999999999e-05 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567143+0000", 2024-09-15T02:48:48.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016539770000000001 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567402+0000", 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000258807 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.570083+0000", 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026806009999999999 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582657+0000", 2024-09-15T02:48:48.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012574469 2024-09-15T02:48:48.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.594754+0000", 2024-09-15T02:48:48.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012097072 2024-09-15T02:48:48.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.594876+0000", 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121605 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:222 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.566500+0000", 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9790662350000003, 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035857158, 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.556 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 222 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566500+0000", 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566500+0000", 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566502+0000", 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9479999999999998e-06 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566986+0000", 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048457099999999999 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567013+0000", 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6429999999999999e-05 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567027+0000", 2024-09-15T02:48:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4246e-05 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578885+0000", 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011858472 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578982+0000", 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6564000000000006e-05 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579827+0000", 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084486900000000005 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583771+0000", 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039440459999999997 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602292+0000", 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018520688 2024-09-15T02:48:48.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602357+0000", 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5324e-05 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:223 3.1s0 3:8610d5f9:::smithi19125844-1:head [setxattr _header (55) in=7b] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567075+0000", 2024-09-15T02:48:48.562 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9784912050000001, 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035541204, 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 223 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567075+0000", 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567075+0000", 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567076+0000", 2024-09-15T02:48:48.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.437e-06 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567083+0000", 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8730000000000001e-06 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567098+0000", 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.484e-05 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567107+0000", 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3540000000000006e-06 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582906+0000", 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015798525000000001 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582986+0000", 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0675999999999998e-05 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583388+0000", 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040186600000000002 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583848+0000", 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046005200000000002 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602517+0000", 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018668657000000002 2024-09-15T02:48:48.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602616+0000", 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8924000000000004e-05 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:224 3.1s0 3:8610d5f9:::smithi19125844-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567193+0000", 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9783733559999996, 2024-09-15T02:48:48.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037648236000000002, 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 224 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567193+0000", 2024-09-15T02:48:48.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567193+0000", 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567194+0000", 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.595e-06 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567199+0000", 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567212+0000", 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2969000000000001e-05 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567221+0000", 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4550000000000001e-06 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583610+0000", 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389053000000001 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583714+0000", 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103657 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.603148+0000", 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019434276 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.603245+0000", 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7081000000000001e-05 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.604841+0000", 2024-09-15T02:48:48.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001595679 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:228 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567230+0000", 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9783360129999998, 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012520063999999999, 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.574 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567230+0000", 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567230+0000", 2024-09-15T02:48:48.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567231+0000", 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4200000000000005e-07 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567235+0000", 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1930000000000003e-06 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567247+0000", 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2177999999999999e-05 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567254+0000", 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2660000000000004e-06 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567332+0000", 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7872000000000003e-05 2024-09-15T02:48:48.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567607+0000", 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027504 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569313+0000", 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001705659 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579381+0000", 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010067606999999999 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579653+0000", 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027204399999999999 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579750+0000", 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7362999999999996e-05 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:229 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567288+0000", 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9782783559999997, 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016039843000000002, 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.580 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 229 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567288+0000", 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567288+0000", 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567288+0000", 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3300000000000002e-07 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567291+0000", 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.858e-06 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567299+0000", 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7749999999999993e-06 2024-09-15T02:48:48.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567303+0000", 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.578e-06 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578451+0000", 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011147631 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578531+0000", 2024-09-15T02:48:48.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0020999999999998e-05 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578935+0000", 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040373999999999998 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579801+0000", 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086611700000000002 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583274+0000", 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003473421 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583327+0000", 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3069000000000003e-05 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:230 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567321+0000", 2024-09-15T02:48:48.585 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9782447330000004, 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017212437000000001, 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 230 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567321+0000", 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567321+0000", 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567322+0000", 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0299999999999999e-07 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567326+0000", 2024-09-15T02:48:48.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5460000000000002e-06 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567333+0000", 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2979999999999997e-06 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567337+0000", 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6330000000000004e-06 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.580164+0000", 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012826671 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.580248+0000", 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4183000000000004e-05 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.580631+0000", 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038259999999999998 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.581376+0000", 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074531499999999997 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.584420+0000", 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030444280000000001 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.584534+0000", 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011326 2024-09-15T02:48:48.590 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 39/33 ECSubWriteReply(tid=91, last_complete=39'24, committed=1, applied=1))", 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.571014+0000", 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9745520670000003, 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083635649999999999, 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571014+0000", 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571014+0000", 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571015+0000", 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1319999999999999e-06 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571021+0000", 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.446e-06 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571079+0000", 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8575000000000001e-05 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571089+0000", 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0387000000000001e-05 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579328+0000", 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082380969999999998 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579377+0000", 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9928e-05 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 39/33 ECSubWriteReply(tid=91, last_complete=39'24, committed=1, applied=1))", 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.571370+0000", 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.974195871, 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084093640000000008, 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571370+0000", 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571370+0000", 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571371+0000", 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.026e-06 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571380+0000", 2024-09-15T02:48:48.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8459999999999997e-06 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571446+0000", 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5594999999999998e-05 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571455+0000", 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5829999999999996e-06 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579565+0000", 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081093280000000007 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579779+0000", 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021498599999999999 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 39/33 ECSubRead(tid=25, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:48.598 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.571526+0000", 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9740399220000002, 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01113599, 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571526+0000", 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571526+0000", 2024-09-15T02:48:48.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571527+0000", 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.06e-07 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571537+0000", 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0158e-05 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571571+0000", 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3958999999999998e-05 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571581+0000", 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.978e-06 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578881+0000", 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073005020000000004 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582662+0000", 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037808870000000001 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.602 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 38/33 ECSubWriteReply(tid=91, last_complete=39'24, committed=1, applied=1))", 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.571971+0000", 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9735947500000002, 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007587908, 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571971+0000", 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571971+0000", 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571972+0000", 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2600000000000001e-07 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571980+0000", 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6260000000000003e-06 2024-09-15T02:48:48.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572034+0000", 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4481000000000002e-05 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572041+0000", 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3899999999999998e-06 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579531+0000", 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074906460000000001 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579559+0000", 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7939000000000001e-05 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.606 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 39/33 ECSubWriteReply(tid=91, last_complete=39'24, committed=1, applied=1))", 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.572535+0000", 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9730313900000001, 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069308950000000003, 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572535+0000", 2024-09-15T02:48:48.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572535+0000", 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572536+0000", 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7060000000000001e-06 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572550+0000", 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3482999999999999e-05 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572639+0000", 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8898000000000002e-05 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572664+0000", 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5639999999999998e-05 2024-09-15T02:48:48.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579449+0000", 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067846950000000003 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579466+0000", 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6473e-05 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 39/33 ECSubRead(tid=26, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.576300+0000", 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9692664180000001, 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065287239999999996, 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576300+0000", 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576300+0000", 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576300+0000", 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7400000000000002e-07 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576313+0000", 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3083e-05 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576371+0000", 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7235999999999999e-05 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576382+0000", 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0889e-05 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582714+0000", 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063324999999999996 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582828+0000", 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011424199999999999 2024-09-15T02:48:48.614 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 39/33 ECSubRead(tid=27, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.576319+0000", 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9692467999999996, 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067125860000000004, 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576319+0000", 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576319+0000", 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576320+0000", 2024-09-15T02:48:48.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9500000000000002e-07 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576326+0000", 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0920000000000003e-06 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576558+0000", 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000232573 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576567+0000", 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0420000000000008e-06 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582890+0000", 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063221509999999998 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583032+0000", 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213299999999999 2024-09-15T02:48:48.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:244 3.7s0 3:e10d9d4d:::smithi19125844-8:head [append 0~688128] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.568905+0000", 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.976661156, 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029261934, 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.568905+0000", 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.568905+0000", 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.568908+0000", 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1329999999999999e-06 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570340+0000", 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014318429999999999 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570380+0000", 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9734999999999998e-05 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570396+0000", 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6019999999999999e-05 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570518+0000", 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012262699999999999 2024-09-15T02:48:48.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570537+0000", 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8433000000000001e-05 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.575595+0000", 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050585049999999996 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.575747+0000", 2024-09-15T02:48:48.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015148000000000001 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577987+0000", 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00223989 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.594217+0000", 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016230531999999999 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.598113+0000", 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038961780000000001 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.598167+0000", 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3557999999999998e-05 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:245 3.7s0 3:e10d9d4d:::smithi19125844-8:head [append 0~376832] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.570836+0000", 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.9747297709999998, 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028185291000000001, 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 245 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570836+0000", 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570836+0000", 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.570838+0000", 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.906e-06 2024-09-15T02:48:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571477+0000", 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00063916600000000004 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571503+0000", 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5265999999999998e-05 2024-09-15T02:48:48.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571512+0000", 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6930000000000002e-06 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571660+0000", 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014771700000000001 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571687+0000", 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7523000000000001e-05 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.588589+0000", 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016901273000000001 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.588735+0000", 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014592399999999999 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589655+0000", 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092073199999999995 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.594219+0000", 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045632379999999998 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.598922+0000", 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047036430000000004 2024-09-15T02:48:48.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.599022+0000", 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9209999999999994e-05 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:246 3.7s0 3:e10d9d4d:::smithi19125844-8:head [setxattr _header (55) in=7b] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.571536+0000", 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.9740297529999999, 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029736419, 2024-09-15T02:48:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 246 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571536+0000", 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571536+0000", 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571538+0000", 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.539e-06 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571549+0000", 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.092e-05 2024-09-15T02:48:48.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571565+0000", 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.662e-05 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571576+0000", 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0879999999999999e-05 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571767+0000", 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191225 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571786+0000", 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8927e-05 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592722+0000", 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020935347999999999 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592924+0000", 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000202019 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.593434+0000", 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051001699999999998 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.594254+0000", 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000820738 2024-09-15T02:48:48.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.601142+0000", 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068874160000000004 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.601273+0000", 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013077 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:247 3.7s0 3:e10d9d4d:::smithi19125844-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.571587+0000", 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.9739788379999998, 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031802587, 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 247 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571587+0000", 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571587+0000", 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571588+0000", 2024-09-15T02:48:48.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.217e-06 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571594+0000", 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7989999999999999e-06 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571611+0000", 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7187e-05 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571622+0000", 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0263e-05 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571832+0000", 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021016900000000001 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.571841+0000", 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6900000000000004e-06 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.593759+0000", 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021917537000000001 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.593875+0000", 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115736 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.601836+0000", 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079613759999999992 2024-09-15T02:48:48.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.601928+0000", 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1736000000000004e-05 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.603390+0000", 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014618770000000001 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 41/33 ECSubWrite(tid=20, reqid=client.4526.0:249, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.591546+0000", 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.954019674, 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020024044000000001, 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.591546+0000", 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.591546+0000", 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.591547+0000", 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3500000000000002e-07 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.591944+0000", 2024-09-15T02:48:48.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039690599999999998 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592017+0000", 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3052000000000005e-05 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592028+0000", 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0892e-05 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592189+0000", 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000161371 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592209+0000", 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0528999999999999e-05 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.611543+0000", 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019333416999999999 2024-09-15T02:48:48.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.611570+0000", 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7542000000000002e-05 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 41/33 ECSubWrite(tid=21, reqid=client.4526.0:250, at_version=41'14, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.604426+0000", 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.941140114, 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084727069999999995, 2024-09-15T02:48:48.649 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604426+0000", 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604426+0000", 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:48.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604427+0000", 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604606+0000", 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017871599999999999 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604671+0000", 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5307000000000003e-05 2024-09-15T02:48:48.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604682+0000", 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1070000000000001e-05 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.607354+0000", 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002672589 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.607366+0000", 2024-09-15T02:48:48.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1419e-05 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612855+0000", 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054887360000000001 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612899+0000", 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3948e-05 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:48.655 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:48.655 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:48.657 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:48:48.658 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:48.653+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:48.658 INFO:tasks.rados.rados.0.smithi191.stdout:110: done (3 left) 2024-09-15T02:48:48.658 INFO:tasks.rados.rados.0.smithi191.stdout:111: read oid 9 snap -1 2024-09-15T02:48:48.658 INFO:tasks.rados.rados.0.smithi191.stdout:111: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-15T02:48:48.658 INFO:tasks.rados.rados.0.smithi191.stdout:112: read oid 22 snap -1 2024-09-15T02:48:48.658 INFO:tasks.rados.rados.0.smithi191.stdout:112: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:48.658 INFO:tasks.rados.rados.0.smithi191.stdout:113: snap_create 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:109: finishing copy_from racing read to smithi19125844-20 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:108: finishing copy_from racing read to smithi19125844-14 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:108: finishing copy_from to smithi19125844-14 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:109: finishing copy_from to smithi19125844-20 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 20 v 18 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 17 (ObjNum 65 snap 4 seq_num 65) dirty exists 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:111: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-15T02:48:48.659 INFO:tasks.rados.rados.0.smithi191.stdout:112: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:48.757 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:48.751+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:48.766 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:48.767 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:48.767 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:48.767 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:48.776 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:48:48.858 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:48.855+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:48.879 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:48.879 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:48.879 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:48.879 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:48.880 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:48.896 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:48:48.959 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:48.951+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:48.997 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:48.997 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:20 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'10, mlcod=24'8)", 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.452670+0000", 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.54178349, 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052649309999999996, 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:48.998 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452670+0000", 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:48.999 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452670+0000", 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452672+0000", 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.906e-06 2024-09-15T02:48:49.000 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452868+0000", 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019606300000000001 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452955+0000", 2024-09-15T02:48:49.001 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6203000000000001e-05 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.452966+0000", 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1521e-05 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.453092+0000", 2024-09-15T02:48:49.002 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012625099999999999 2024-09-15T02:48:49.003 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.003 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.003 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.003 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.453158+0000", 2024-09-15T02:48:49.003 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5332000000000004e-05 2024-09-15T02:48:49.003 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.004 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.004 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.004 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.457827+0000", 2024-09-15T02:48:49.004 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0046690009999999999 2024-09-15T02:48:49.004 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.004 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.457935+0000", 2024-09-15T02:48:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010865399999999999 2024-09-15T02:48:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.006 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.006 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.006 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:41 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'23, mlcod=24'19)", 2024-09-15T02:48:49.006 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.729948+0000", 2024-09-15T02:48:49.006 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.264505446000001, 2024-09-15T02:48:49.006 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052060489999999999, 2024-09-15T02:48:49.007 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.007 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.007 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.007 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.007 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.007 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.729948+0000", 2024-09-15T02:48:49.008 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.008 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.008 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.008 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.008 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.729948+0000", 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.729949+0000", 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1999999999999999e-07 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.009 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730305+0000", 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00035559099999999999 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730470+0000", 2024-09-15T02:48:49.010 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000165012 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730504+0000", 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4325000000000001e-05 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.011 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730828+0000", 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032381700000000001 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.012 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730856+0000", 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7864999999999998e-05 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735100+0000", 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042445809999999999 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.013 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735155+0000", 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4138000000000002e-05 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:46 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'28, mlcod=24'19)", 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.264036905000001, 2024-09-15T02:48:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070834360000000002, 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.015 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730417+0000", 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5999999999999999e-07 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730421+0000", 2024-09-15T02:48:49.016 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8770000000000003e-06 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730887+0000", 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000465253 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.017 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730893+0000", 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7100000000000001e-06 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733567+0000", 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002673922 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733619+0000", 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1465000000000003e-05 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.018 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.737397+0000", 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0037785319999999998 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.737500+0000", 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000103317 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.019 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:47 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'29, mlcod=24'19)", 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.730800+0000", 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.263654108000001, 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0072831750000000002, 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.020 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730800+0000", 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730800+0000", 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.021 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730801+0000", 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.176e-06 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.730850+0000", 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8541999999999997e-05 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732102+0000", 2024-09-15T02:48:49.022 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012526810000000001 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732118+0000", 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6198e-05 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734170+0000", 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0020519150000000001 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.023 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734198+0000", 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7614e-05 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738021+0000", 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0038232800000000001 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738083+0000", 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.1768999999999998e-05 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.024 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:48 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'30, mlcod=24'20)", 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.732043+0000", 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.262410804000002, 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060923130000000002, 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.025 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732043+0000", 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.026 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732043+0000", 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732045+0000", 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6199999999999999e-06 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732060+0000", 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5387999999999999e-05 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.027 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732930+0000", 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00086952500000000001 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732943+0000", 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2947e-05 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.028 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734641+0000", 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016985640000000001 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.734674+0000", 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2977e-05 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738088+0000", 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0034136269999999998 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.029 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738135+0000", 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7664999999999998e-05 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:49 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'31, mlcod=24'20)", 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.732838+0000", 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.261616004, 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053466640000000001, 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.030 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732838+0000", 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732838+0000", 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.031 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732840+0000", 2024-09-15T02:48:49.032 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:48:49.032 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.032 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.032 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.032 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.732879+0000", 2024-09-15T02:48:49.033 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9944999999999998e-05 2024-09-15T02:48:49.033 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.033 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.033 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.033 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733806+0000", 2024-09-15T02:48:49.033 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00092643399999999996 2024-09-15T02:48:49.034 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.034 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.034 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.034 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.733840+0000", 2024-09-15T02:48:49.034 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4187999999999999e-05 2024-09-15T02:48:49.034 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.035 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.035 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.035 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735397+0000", 2024-09-15T02:48:49.035 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0015569780000000001 2024-09-15T02:48:49.035 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.035 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.036 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.036 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.735431+0000", 2024-09-15T02:48:49.036 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4381000000000002e-05 2024-09-15T02:48:49.036 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.036 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.036 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738139+0000", 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0027073539999999999 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.738185+0000", 2024-09-15T02:48:49.037 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:48:49.038 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.038 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.038 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.038 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.038 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.038 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:48:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.258173159999998, 2024-09-15T02:48:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:48:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.040 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.040 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.040 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:49.040 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.040 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.040 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.041 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.041 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:49.041 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.041 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.041 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.041 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.042 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:48:49.042 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:48:49.042 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.042 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.042 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.043 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:48:49.043 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:48:49.043 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.043 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.043 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.043 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:48:49.044 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:48:49.044 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.044 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.044 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.045 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:48:49.045 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:48:49.045 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.045 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.045 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.045 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:48:49.046 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:48:49.046 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.046 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.046 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.046 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:48:49.046 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:48:49.047 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.047 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.047 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.047 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:48:49.047 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:48:49.048 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.048 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.048 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.048 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:48:49.048 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:48:49.048 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.049 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.049 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.049 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.049 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.049 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:71 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'49, mlcod=24'45)", 2024-09-15T02:48:49.049 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.339997+0000", 2024-09-15T02:48:49.050 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.654456932999999, 2024-09-15T02:48:49.050 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0065061889999999999, 2024-09-15T02:48:49.050 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.050 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.050 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.050 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.051 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.051 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.339997+0000", 2024-09-15T02:48:49.051 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.051 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.051 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.051 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.052 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.339997+0000", 2024-09-15T02:48:49.052 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.052 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.052 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.052 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.053 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.339999+0000", 2024-09-15T02:48:49.053 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6199999999999999e-06 2024-09-15T02:48:49.053 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.053 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.053 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.053 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340011+0000", 2024-09-15T02:48:49.054 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2801e-05 2024-09-15T02:48:49.054 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.054 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.054 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.054 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340091+0000", 2024-09-15T02:48:49.054 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0065e-05 2024-09-15T02:48:49.055 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.055 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.055 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.055 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340102+0000", 2024-09-15T02:48:49.055 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0484e-05 2024-09-15T02:48:49.056 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.056 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.056 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.056 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340192+0000", 2024-09-15T02:48:49.056 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0513000000000006e-05 2024-09-15T02:48:49.056 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.057 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.057 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.057 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340240+0000", 2024-09-15T02:48:49.057 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7845e-05 2024-09-15T02:48:49.057 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.057 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.058 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.058 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346425+0000", 2024-09-15T02:48:49.058 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0061847940000000004 2024-09-15T02:48:49.058 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.058 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.059 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.059 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346503+0000", 2024-09-15T02:48:49.059 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8066999999999996e-05 2024-09-15T02:48:49.059 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.059 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.059 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.060 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.060 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.060 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:72 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'50, mlcod=24'46)", 2024-09-15T02:48:49.060 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.340749+0000", 2024-09-15T02:48:49.060 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.653704749999999, 2024-09-15T02:48:49.060 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058094349999999999, 2024-09-15T02:48:49.061 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.061 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.061 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.061 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.061 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.062 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340749+0000", 2024-09-15T02:48:49.062 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.062 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.062 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.062 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.062 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340749+0000", 2024-09-15T02:48:49.063 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.063 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.063 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.063 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.063 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340751+0000", 2024-09-15T02:48:49.063 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.347e-06 2024-09-15T02:48:49.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.064 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.064 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.064 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340808+0000", 2024-09-15T02:48:49.064 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7008e-05 2024-09-15T02:48:49.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.065 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.065 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.065 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340873+0000", 2024-09-15T02:48:49.065 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5611999999999995e-05 2024-09-15T02:48:49.065 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.066 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.066 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.066 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340885+0000", 2024-09-15T02:48:49.066 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.15e-05 2024-09-15T02:48:49.066 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.066 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.067 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.067 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.340964+0000", 2024-09-15T02:48:49.067 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9312000000000002e-05 2024-09-15T02:48:49.067 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.067 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.067 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.068 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.341002+0000", 2024-09-15T02:48:49.068 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8331999999999998e-05 2024-09-15T02:48:49.068 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.068 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.068 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.069 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346514+0000", 2024-09-15T02:48:49.069 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055117480000000003 2024-09-15T02:48:49.069 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.069 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.069 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.069 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346559+0000", 2024-09-15T02:48:49.070 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4576000000000001e-05 2024-09-15T02:48:49.070 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.070 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.070 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.070 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.070 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.071 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:73 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'51, mlcod=24'46)", 2024-09-15T02:48:49.071 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.342123+0000", 2024-09-15T02:48:49.071 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.652330436, 2024-09-15T02:48:49.071 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070136319999999997, 2024-09-15T02:48:49.071 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.072 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.072 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.072 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.072 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.072 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342123+0000", 2024-09-15T02:48:49.073 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.073 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.073 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.073 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.073 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342123+0000", 2024-09-15T02:48:49.073 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.074 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.074 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.074 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.074 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342125+0000", 2024-09-15T02:48:49.074 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.254e-06 2024-09-15T02:48:49.075 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.075 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.075 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.075 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342137+0000", 2024-09-15T02:48:49.075 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2027e-05 2024-09-15T02:48:49.075 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.076 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.076 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.076 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342206+0000", 2024-09-15T02:48:49.076 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9605999999999997e-05 2024-09-15T02:48:49.076 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.077 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.077 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.077 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342216+0000", 2024-09-15T02:48:49.077 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3899999999999999e-06 2024-09-15T02:48:49.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345367+0000", 2024-09-15T02:48:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0031511669999999999 2024-09-15T02:48:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.079 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.079 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345413+0000", 2024-09-15T02:48:49.079 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6516e-05 2024-09-15T02:48:49.079 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.080 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.080 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.080 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349036+0000", 2024-09-15T02:48:49.080 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036222989999999998 2024-09-15T02:48:49.080 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.081 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.081 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.081 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349137+0000", 2024-09-15T02:48:49.081 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010137300000000001 2024-09-15T02:48:49.081 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.082 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.082 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.082 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.082 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.082 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:74 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'52, mlcod=24'47)", 2024-09-15T02:48:49.082 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.342890+0000", 2024-09-15T02:48:49.083 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.651563505999999, 2024-09-15T02:48:49.083 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0063358019999999998, 2024-09-15T02:48:49.083 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.083 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.083 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.083 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.084 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.084 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342890+0000", 2024-09-15T02:48:49.084 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.084 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.084 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.084 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.085 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342890+0000", 2024-09-15T02:48:49.085 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.085 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.085 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.085 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.085 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342891+0000", 2024-09-15T02:48:49.086 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0999999999999998e-07 2024-09-15T02:48:49.086 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.086 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.086 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.086 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342899+0000", 2024-09-15T02:48:49.087 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9580000000000007e-06 2024-09-15T02:48:49.087 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.087 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.087 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.087 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.342967+0000", 2024-09-15T02:48:49.087 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8088000000000003e-05 2024-09-15T02:48:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343024+0000", 2024-09-15T02:48:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6577999999999999e-05 2024-09-15T02:48:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.089 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.089 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.089 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345902+0000", 2024-09-15T02:48:49.089 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028785220000000001 2024-09-15T02:48:49.089 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.089 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.090 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.090 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.345946+0000", 2024-09-15T02:48:49.090 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3603999999999998e-05 2024-09-15T02:48:49.090 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.090 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.090 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349145+0000", 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003198986 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349226+0000", 2024-09-15T02:48:49.091 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1355999999999998e-05 2024-09-15T02:48:49.092 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.092 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.092 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.092 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.092 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.092 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'47)", 2024-09-15T02:48:49.093 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.343614+0000", 2024-09-15T02:48:49.093 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.650840393999999, 2024-09-15T02:48:49.093 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0059812909999999997, 2024-09-15T02:48:49.093 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.093 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.093 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.094 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.094 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.094 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343614+0000", 2024-09-15T02:48:49.094 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.094 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.094 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.095 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.095 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343614+0000", 2024-09-15T02:48:49.095 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.095 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.095 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.095 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.096 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343615+0000", 2024-09-15T02:48:49.096 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.257e-06 2024-09-15T02:48:49.096 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.096 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.096 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.096 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343689+0000", 2024-09-15T02:48:49.097 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4176000000000003e-05 2024-09-15T02:48:49.097 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.097 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.097 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.097 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343772+0000", 2024-09-15T02:48:49.097 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3279e-05 2024-09-15T02:48:49.098 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.098 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.098 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.098 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.343781+0000", 2024-09-15T02:48:49.098 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0559999999999994e-06 2024-09-15T02:48:49.098 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.099 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.099 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.099 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346598+0000", 2024-09-15T02:48:49.099 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028166129999999999 2024-09-15T02:48:49.099 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.099 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.100 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.100 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.346631+0000", 2024-09-15T02:48:49.100 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3498999999999999e-05 2024-09-15T02:48:49.100 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.100 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.101 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.101 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349465+0000", 2024-09-15T02:48:49.101 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002833534 2024-09-15T02:48:49.101 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.101 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.101 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.102 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.349595+0000", 2024-09-15T02:48:49.102 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000129877 2024-09-15T02:48:49.102 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.102 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.102 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:99 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'73, mlcod=25'66)", 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:25.579821+0000", 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.414632498, 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055442140000000004, 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.103 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579821+0000", 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579821+0000", 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579823+0000", 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0899999999999999e-06 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.104 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579938+0000", 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000115767 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.579991+0000", 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2797999999999998e-05 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.580001+0000", 2024-09-15T02:48:49.105 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0003999999999999e-05 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.580473+0000", 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00047201200000000001 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.580489+0000", 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6399e-05 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.106 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.585327+0000", 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048373829999999998 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.585366+0000", 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8760999999999997e-05 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:137 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'103, mlcod=25'100)", 2024-09-15T02:48:49.107 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:28.314615+0000", 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 20.679839117, 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053880100000000004, 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314615+0000", 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.108 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314615+0000", 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314616+0000", 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.305e-06 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314630+0000", 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3436e-05 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.109 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314695+0000", 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5398e-05 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314703+0000", 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3760000000000008e-06 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314809+0000", 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010540900000000001 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.110 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.314854+0000", 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5571999999999997e-05 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.319943+0000", 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050882210000000004 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:28.320003+0000", 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0293e-05 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.111 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:220 3.es0 3:71c1a91c:::smithi19125844-45:head [create,setxattr YOk\\Qsqnjk`Uh[icZasWhcZnad[h[k[Qmk]TihbR (1823) in=40b,setxattr QnfshqY (34) in=7b,setxattr V_tZOrZWhpPekPNs_toQaraRlW\\sWffpapfPROXb (1614) in=40b,setxattr aiTbV]hXcpXjkS]VQonZZkfogbUSnO[bajbZWdRf (839) in=40b,setxattr pZdlecZrV]qp]`sTkf_bSjZVZjhbScXfb[Sr_bsa (18524) in=40b,setxattr kXtOdjofgYOa_Tck\\nYsbVOfgtldphnQ`]P[j]lX (1295) in=40b,setxattr dRpscRhc^OtlbThWSmXVSNge]s (34) in=26b,setxattr T\\s[QdNhlRZh\\\\[X]NpSccmfRcqRaXScZOZ\\ZR^e (1999) in=40b,setxattr YTcfnqlbrRa^rhctrncU_Wa]laamPmYsaqg_f^N` (1471) in=40b,setxattr l_Qeh\\RcafrRYN_YZf_iiaWagZfe[sop]^\\qTggq (561) in=40b,setxattr NShVOQe\\dRYY[tPettNbkgkTVonNshnbYeePWcla (314) in=40b,setxattr ^iVidVQX]Yn`XTY\\pOXm`_PNX`qeckhbQoR`aQWZ (8841) in=40b,setxattr _jkmO]cXrN_S^act\\QOOjpUZOj_Qf^otnZbpWZPU (981) in=40b,setxattr bNbhPX[YP[lppmc`PkWcpnS`XsQNpW\\UYnWr\\_RX (1927) in=40b,setxattr talTebhpYS[[TXcgodoQR\\tf`fVekppjlkWnTqf[ (1753) in=40b,setxattr rp\\XTgd\\X^qaodkhY[elq^RagRhN_NR^j^NmjOUh (806) in=40b,setxattr ]\\oXSo`PWafUO^dbhR^i`hifQgWjRhR[rRa^bnej (70) in=40b,setxattr UkYrdYaZakpnReakOdc\\^bO[YPqfdkdtbNkghaSQ (279) in=40b,setxattr YtUs_hcRdQqgZV]NaadOflcb\\_c^\\Zg_Y\\]snXXZ (701) in=40b,setxattr NRZdh]WclahTe^_btdqe\\^mVYqQOaXaa\\aPVXWnk (19016) in=40b] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.554086+0000", 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.4403675380000003, 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0098341899999999996, 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:49.112 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 220 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554086+0000", 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554086+0000", 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.113 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554089+0000", 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5670000000000002e-06 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554222+0000", 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013293199999999999 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554249+0000", 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7223e-05 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.114 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554262+0000", 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3285000000000001e-05 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554370+0000", 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010724499999999999 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554397+0000", 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7356999999999999e-05 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.115 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.558117+0000", 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00372048 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.558280+0000", 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016244400000000001 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560050+0000", 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017700280000000001 2024-09-15T02:48:49.116 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.561521+0000", 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001470741 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.563801+0000", 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002279934 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.563921+0000", 2024-09-15T02:48:49.117 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000119954 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:226 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565176+0000", 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.4292783980000001, 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00595956, 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:49.118 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 226 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565176+0000", 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565176+0000", 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.119 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565177+0000", 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9699999999999994e-07 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565182+0000", 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5729999999999998e-06 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565197+0000", 2024-09-15T02:48:49.120 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4462e-05 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565206+0000", 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2930000000000006e-06 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568061+0000", 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028554679999999999 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.121 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568161+0000", 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9857000000000003e-05 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571135+0000", 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0029739100000000002 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:227 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565226+0000", 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.4292276380000004, 2024-09-15T02:48:49.122 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0064754239999999996, 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 227 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565226+0000", 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565226+0000", 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565228+0000", 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2610000000000001e-06 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565233+0000", 2024-09-15T02:48:49.124 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5550000000000002e-06 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565246+0000", 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2887999999999999e-05 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565253+0000", 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.72e-06 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.125 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569974+0000", 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0047212349999999998 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570074+0000", 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9813000000000001e-05 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571702+0000", 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001627952 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.126 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.568541+0000", 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.4259128150000002, 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013583105, 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.127 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568541+0000", 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568541+0000", 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568542+0000", 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3099999999999996e-07 2024-09-15T02:48:49.128 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570537+0000", 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0019946109999999999 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570648+0000", 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000111309 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.129 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570656+0000", 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5569999999999996e-06 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570713+0000", 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6875e-05 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570722+0000", 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8349999999999993e-06 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.130 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582100+0000", 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011377395 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582124+0000", 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4692e-05 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.131 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.2s1 44/33 ECSubWrite(tid=161, reqid=osd.11.0:161, at_version=44'21, trim_to=0'0, roll_forward_to=43'17))", 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.606441+0000", 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.38801244299999998, 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053102760000000001, 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606441+0000", 2024-09-15T02:48:49.132 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606441+0000", 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606443+0000", 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.172e-06 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.133 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606489+0000", 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6249999999999999e-05 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606591+0000", 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010237400000000001 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606603+0000", 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2108e-05 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.134 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606664+0000", 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0891999999999999e-05 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606673+0000", 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3669999999999997e-06 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.611715+0000", 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.005042052 2024-09-15T02:48:49.135 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.611752+0000", 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7061000000000003e-05 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs1 44/33 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.607803+0000", 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.386650985, 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0066622970000000002, 2024-09-15T02:48:49.136 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607803+0000", 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607803+0000", 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.137 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607803+0000", 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4699999999999998e-07 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607832+0000", 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8863000000000001e-05 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607922+0000", 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9931000000000003e-05 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.138 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607935+0000", 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2622999999999999e-05 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607999+0000", 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.4733000000000005e-05 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608007+0000", 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8439999999999998e-06 2024-09-15T02:48:49.139 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614431+0000", 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0064238189999999999 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614465+0000", 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4236999999999998e-05 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.140 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.141 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:49.142 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:48:49.142 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:49.135+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:49.243 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:49.237+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:49.247 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:49.247 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:49.247 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:49.247 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:49.256 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:48:49.344 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:49.341+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:49.357 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:49.357 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:49.358 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:49.358 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:49.358 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:49.366 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:48:49.445 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:49.441+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:49.471 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:49.471 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.es5 39/33 ECSubWrite(tid=27, reqid=client.4526.0:220, at_version=39'14, trim_to=0'0, roll_forward_to=36'13))", 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.560211+0000", 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.9071023949999999, 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016019879999999999, 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.472 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560211+0000", 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560211+0000", 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560211+0000", 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3000000000000001e-07 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560335+0000", 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000123313 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.474 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560443+0000", 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000108256 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560459+0000", 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6169999999999999e-05 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560530+0000", 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0789000000000003e-05 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560541+0000", 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1545999999999999e-05 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.561750+0000", 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001208813 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.561813+0000", 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2570999999999994e-05 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 39/33 ECSubWrite(tid=91, reqid=client.4526.0:228, at_version=39'24, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.568301+0000", 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.8990120600000004, 2024-09-15T02:48:49.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013560149999999999, 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568301+0000", 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568301+0000", 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568302+0000", 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568352+0000", 2024-09-15T02:48:49.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9839999999999997e-05 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568429+0000", 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7312999999999997e-05 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568440+0000", 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1124000000000001e-05 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.480 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568524+0000", 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3714999999999994e-05 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568542+0000", 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8773000000000001e-05 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569615+0000", 2024-09-15T02:48:49.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010728070000000001 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569657+0000", 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1993000000000003e-05 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.483 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.5s6 39/33 ECSubWrite(tid=16, reqid=client.4526.0:231, at_version=39'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:49.483 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.571289+0000", 2024-09-15T02:48:49.483 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.8960241470000003, 2024-09-15T02:48:49.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0015717490000000001, 2024-09-15T02:48:49.483 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.483 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571289+0000", 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.485 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571289+0000", 2024-09-15T02:48:49.485 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.485 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.485 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.485 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571290+0000", 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571324+0000", 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4629999999999999e-05 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571378+0000", 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4191999999999999e-05 2024-09-15T02:48:49.486 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571388+0000", 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.4120000000000007e-06 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571470+0000", 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2408000000000001e-05 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571491+0000", 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0560999999999998e-05 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572802+0000", 2024-09-15T02:48:49.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001311354 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572861+0000", 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8811999999999999e-05 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.5s6 39/33 ECSubWrite(tid=18, reqid=client.4526.0:233, at_version=39'11, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:49.489 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.574117+0000", 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.8931966490000001, 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0024618420000000001, 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574117+0000", 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574117+0000", 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574117+0000", 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999995e-07 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574136+0000", 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8989999999999999e-05 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.491 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574172+0000", 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6133000000000001e-05 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.492 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574179+0000", 2024-09-15T02:48:49.493 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8229999999999997e-06 2024-09-15T02:48:49.493 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574251+0000", 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1846000000000003e-05 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.574259+0000", 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1149999999999994e-06 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576540+0000", 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022808379999999999 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.576578+0000", 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8368999999999998e-05 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 39/33 ECSubWrite(tid=92, reqid=client.4526.0:229, at_version=39'25, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.577725+0000", 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.8895885320000003, 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017796229999999999, 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577725+0000", 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.496 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577725+0000", 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577726+0000", 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.4e-07 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.497 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577751+0000", 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5173000000000001e-05 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577807+0000", 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6385000000000003e-05 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.498 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577833+0000", 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5558000000000002e-05 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577892+0000", 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9438999999999998e-05 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.577904+0000", 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1817000000000001e-05 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.499 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.579456+0000", 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0015523500000000001 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.579504+0000", 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7960999999999997e-05 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.500 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 39/33 ECSubWrite(tid=96, reqid=client.4526.0:230, at_version=0'0, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.583461+0000", 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.8838526470000003, 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0015791430000000001, 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583461+0000", 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583461+0000", 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583461+0000", 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5600000000000004e-07 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.502 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583472+0000", 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0028e-05 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583501+0000", 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9077e-05 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583506+0000", 2024-09-15T02:48:49.503 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8109999999999999e-06 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583564+0000", 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7500000000000002e-05 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.583573+0000", 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8599999999999999e-06 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.504 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.585004+0000", 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0014313430000000001 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.585040+0000", 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5667999999999998e-05 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.5s6 39/33 ECSubWrite(tid=19, reqid=client.4526.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:49.505 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.056215+0000", 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.4110987550000003, 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018143300000000001, 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056215+0000", 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.506 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056215+0000", 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056217+0000", 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1459999999999999e-06 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.507 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056244+0000", 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7537000000000001e-05 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056333+0000", 2024-09-15T02:48:49.508 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8514e-05 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056347+0000", 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4354e-05 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.509 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056499+0000", 2024-09-15T02:48:49.510 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015185099999999999 2024-09-15T02:48:49.510 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.510 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.510 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.510 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.056529+0000", 2024-09-15T02:48:49.511 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9777e-05 2024-09-15T02:48:49.511 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.511 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.511 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.511 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.057957+0000", 2024-09-15T02:48:49.511 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001428014 2024-09-15T02:48:49.512 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.512 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.512 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.512 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.058029+0000", 2024-09-15T02:48:49.512 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2137000000000004e-05 2024-09-15T02:48:49.512 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.577346+0000", 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.8899669440000002, 2024-09-15T02:48:49.513 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019179986, 2024-09-15T02:48:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577346+0000", 2024-09-15T02:48:49.515 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.515 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.515 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.515 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.515 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577346+0000", 2024-09-15T02:48:49.515 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.516 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.516 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.516 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.516 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577347+0000", 2024-09-15T02:48:49.516 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.35e-07 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577748+0000", 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00040174200000000002 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.517 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577791+0000", 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2456999999999997e-05 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577804+0000", 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.518 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577967+0000", 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016357899999999999 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577985+0000", 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7880000000000002e-05 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596468+0000", 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018483133999999998 2024-09-15T02:48:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596526+0000", 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7969e-05 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=100, reqid=client.4526.0:245, at_version=40'29, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.588588+0000", 2024-09-15T02:48:49.520 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.8787255099999998, 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087388440000000008, 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588588+0000", 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.521 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588588+0000", 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588590+0000", 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3149999999999999e-06 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588712+0000", 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012221000000000001 2024-09-15T02:48:49.522 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588784+0000", 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1898999999999997e-05 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588795+0000", 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0878e-05 2024-09-15T02:48:49.523 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.590216+0000", 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001420803 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.590233+0000", 2024-09-15T02:48:49.524 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6875e-05 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.597294+0000", 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070608559999999999 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.597327+0000", 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3008e-05 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.525 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=101, reqid=client.4526.0:246, at_version=40'30, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.592222+0000", 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.8750913929999999, 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0073089950000000004, 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592222+0000", 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.526 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592222+0000", 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592224+0000", 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2139999999999999e-06 2024-09-15T02:48:49.527 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.528 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.528 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.528 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592261+0000", 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6436000000000001e-05 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592328+0000", 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7535999999999998e-05 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592348+0000", 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0080000000000001e-05 2024-09-15T02:48:49.529 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596634+0000", 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042860229999999999 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596664+0000", 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9774e-05 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.599492+0000", 2024-09-15T02:48:49.530 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028282899999999998 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.599531+0000", 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8642e-05 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=102, reqid=client.4526.0:246, at_version=0'0, trim_to=0'0, roll_forward_to=40'30))", 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.600282+0000", 2024-09-15T02:48:49.531 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.8670309730000003, 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012369480000000001, 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600282+0000", 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.532 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600282+0000", 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600284+0000", 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1579999999999999e-06 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.533 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600314+0000", 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9311000000000001e-05 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.534 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600372+0000", 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8723e-05 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600392+0000", 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9283000000000002e-05 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600482+0000", 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9765000000000005e-05 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.536 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.600501+0000", 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9247000000000001e-05 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.601494+0000", 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00099332099999999992 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.601519+0000", 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.514e-05 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.537 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.5s6 44/33 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=44'21, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.596878+0000", 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.87043567700000002, 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003001456, 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.538 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.596878+0000", 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.596878+0000", 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.596878+0000", 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3000000000000001e-07 2024-09-15T02:48:49.539 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.596921+0000", 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3133000000000002e-05 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.597013+0000", 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2276999999999998e-05 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.540 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.597038+0000", 2024-09-15T02:48:49.541 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4133000000000001e-05 2024-09-15T02:48:49.541 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.541 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.541 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.541 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.597151+0000", 2024-09-15T02:48:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000113884 2024-09-15T02:48:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.597170+0000", 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8783999999999998e-05 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.599833+0000", 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026626670000000001 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.599879+0000", 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6147999999999999e-05 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.5s6 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.598690+0000", 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.868623586, 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0020339609999999999, 2024-09-15T02:48:49.544 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.598690+0000", 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.598690+0000", 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.545 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.598691+0000", 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6300000000000001e-06 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.598749+0000", 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8010000000000002e-05 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.546 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.598838+0000", 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8187000000000001e-05 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.598855+0000", 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7326e-05 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.547 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.548 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.548 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.599038+0000", 2024-09-15T02:48:49.548 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018282500000000001 2024-09-15T02:48:49.548 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.549 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.549 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.549 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.599064+0000", 2024-09-15T02:48:49.549 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.654e-05 2024-09-15T02:48:49.550 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.550 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.550 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.550 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.600645+0000", 2024-09-15T02:48:49.550 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001581177 2024-09-15T02:48:49.550 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.600724+0000", 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8265999999999999e-05 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.551 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.552 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.552 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.7s1 44/33 ECSubRead(tid=116, to_read={3:e1027d4a:::smithi19125844-9:head=0,253952,0}, subchunks={3:e1027d4a:::smithi19125844-9:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:49.552 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.605420+0000", 2024-09-15T02:48:49.552 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.86189375599999996, 2024-09-15T02:48:49.552 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013575729999999999, 2024-09-15T02:48:49.552 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.553 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.553 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.553 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.553 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.553 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.605420+0000", 2024-09-15T02:48:49.553 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.554 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.554 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.554 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.554 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.605420+0000", 2024-09-15T02:48:49.554 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.554 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.605421+0000", 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.263e-06 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.555 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.605437+0000", 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6478e-05 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606219+0000", 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00078181400000000003 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606236+0000", 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7002000000000001e-05 2024-09-15T02:48:49.556 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606504+0000", 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00026771199999999997 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.606777+0000", 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027330399999999998 2024-09-15T02:48:49.557 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/33 ECSubWrite(tid=117, reqid=osd.13.0:117, at_version=44'40, trim_to=0'0, roll_forward_to=43'38))", 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.607074+0000", 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.86023932199999997, 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043681190000000002, 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.558 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607074+0000", 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607074+0000", 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607075+0000", 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.026e-06 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.559 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607144+0000", 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8999999999999997e-05 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607430+0000", 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028606399999999999 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.560 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607439+0000", 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0799999999999995e-06 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607508+0000", 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8929000000000004e-05 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607517+0000", 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1409999999999994e-06 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.561 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.611405+0000", 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0038877849999999999 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.611442+0000", 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7094000000000001e-05 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.562 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.5s6 44/33 ECSubWrite(tid=31, reqid=osd.8.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.608281+0000", 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.85903235099999997, 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0061270960000000003, 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608281+0000", 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.563 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608281+0000", 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608282+0000", 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1799999999999999e-06 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608298+0000", 2024-09-15T02:48:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6059999999999999e-05 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608388+0000", 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0091000000000002e-05 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608397+0000", 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1740000000000006e-06 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.565 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608477+0000", 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9418000000000004e-05 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608488+0000", 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1408e-05 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614363+0000", 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058743720000000001 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.566 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614408+0000", 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5392999999999997e-05 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/33 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=44'42, trim_to=0'0, roll_forward_to=43'38))", 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.608786+0000", 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.85852756900000005, 2024-09-15T02:48:49.567 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056028939999999998, 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608786+0000", 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608786+0000", 2024-09-15T02:48:49.568 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608787+0000", 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.689e-06 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608832+0000", 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4325e-05 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608902+0000", 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9882999999999995e-05 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608917+0000", 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4982e-05 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.609022+0000", 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000104973 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.570 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.609046+0000", 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4717000000000001e-05 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614336+0000", 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052902110000000004 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614389+0000", 2024-09-15T02:48:49.571 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2114000000000003e-05 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.as5 44/33 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=44'15, trim_to=0'0, roll_forward_to=43'13))", 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.85706364199999996, 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070532809999999998, 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.572 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.573 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.17e-07 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610273+0000", 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3404000000000002e-05 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.574 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610322+0000", 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8609000000000002e-05 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610329+0000", 2024-09-15T02:48:49.575 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8600000000000004e-06 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610427+0000", 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8215999999999996e-05 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610447+0000", 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0149999999999999e-05 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.576 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617264+0000", 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0068173050000000001 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617303+0000", 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8519999999999997e-05 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.577 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.as5 44/33 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618369+0000", 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.84894468099999998, 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017095140000000001, 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618369+0000", 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.578 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618369+0000", 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618370+0000", 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5689999999999999e-06 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.579 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618401+0000", 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0434e-05 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618454+0000", 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3863000000000003e-05 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618462+0000", 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9629999999999998e-06 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618544+0000", 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1143000000000006e-05 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618560+0000", 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6929e-05 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620039+0000", 2024-09-15T02:48:49.581 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001478566 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620078+0000", 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9047000000000001e-05 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.582 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.583 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.583 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.583 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/33 ECSubWrite(tid=120, reqid=osd.13.0:118, at_version=0'0, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:49.583 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.620099+0000", 2024-09-15T02:48:49.583 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.84721397499999995, 2024-09-15T02:48:49.583 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030446570000000001, 2024-09-15T02:48:49.584 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:49.584 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:49.584 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:49.584 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.584 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:49.584 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620099+0000", 2024-09-15T02:48:49.585 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.585 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.585 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.585 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:49.585 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620099+0000", 2024-09-15T02:48:49.585 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:49.586 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.586 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.586 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:49.586 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620100+0000", 2024-09-15T02:48:49.586 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1629999999999999e-06 2024-09-15T02:48:49.586 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620117+0000", 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6364000000000001e-05 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:49.587 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620177+0000", 2024-09-15T02:48:49.588 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0520999999999999e-05 2024-09-15T02:48:49.588 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.588 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.588 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:49.588 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620187+0000", 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9909999999999997e-06 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620276+0000", 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8200000000000003e-05 2024-09-15T02:48:49.589 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.590 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.590 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:49.590 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620291+0000", 2024-09-15T02:48:49.590 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5576000000000001e-05 2024-09-15T02:48:49.590 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.590 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.623098+0000", 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002807019 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.623144+0000", 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5822999999999998e-05 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.591 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.592 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:49.592 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:49.592 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:49.593 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:48:49.594 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:49.590+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:49.615 INFO:tasks.rados.rados.0.smithi191.stdout:107: done (5 left) 2024-09-15T02:48:49.615 INFO:tasks.rados.rados.0.smithi191.stdout:108: done (4 left) 2024-09-15T02:48:49.615 INFO:tasks.rados.rados.0.smithi191.stdout:109: done (3 left) 2024-09-15T02:48:49.615 INFO:tasks.rados.rados.0.smithi191.stdout:111: done (2 left) 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:112: done (1 left) 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:113: done (0 left) 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:114: read oid 22 snap -1 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:114: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:115: rmattr oid 16 current snap is 5 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:115: done (1 left) 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:116: read oid 21 snap 4 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:116: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T02:48:49.616 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 5 current snap is 5 2024-09-15T02:48:49.617 INFO:tasks.rados.rados.0.smithi191.stdout:117: seq_num 68 ranges {933888=770048,1703936=376832} 2024-09-15T02:48:49.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:48:49.631 INFO:tasks.rados.rados.0.smithi191.stdout:117: writing smithi19125844-5 from 933888 to 1703936 tid 1 2024-09-15T02:48:49.633 INFO:tasks.rados.rados.0.smithi191.stdout:117: writing smithi19125844-5 from 1703936 to 2080768 tid 2 2024-09-15T02:48:49.633 INFO:tasks.rados.rados.0.smithi191.stdout:118: read oid 26 snap 4 2024-09-15T02:48:49.633 INFO:tasks.rados.rados.0.smithi191.stdout:118: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-15T02:48:49.633 INFO:tasks.rados.rados.0.smithi191.stdout:119: delete oid 45 current snap is 5 2024-09-15T02:48:49.633 INFO:tasks.rados.rados.0.smithi191.stdout:116: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T02:48:49.648 INFO:tasks.rados.rados.0.smithi191.stdout:116: done (4 left) 2024-09-15T02:48:49.648 INFO:tasks.rados.rados.0.smithi191.stdout:119: done (3 left) 2024-09-15T02:48:49.648 INFO:tasks.rados.rados.0.smithi191.stdout:120: snap_create 2024-09-15T02:48:49.648 INFO:tasks.rados.rados.0.smithi191.stdout:114: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:49.658 INFO:tasks.rados.rados.0.smithi191.stdout:118: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-15T02:48:49.671 INFO:tasks.rados.rados.0.smithi191.stdout:117: finishing write tid 1 to smithi19125844-5 2024-09-15T02:48:49.671 INFO:tasks.rados.rados.0.smithi191.stdout:117: finishing write tid 2 to smithi19125844-5 2024-09-15T02:48:49.671 INFO:tasks.rados.rados.0.smithi191.stdout:117: finishing write tid 3 to smithi19125844-5 2024-09-15T02:48:49.671 INFO:tasks.rados.rados.0.smithi191.stdout:117: finishing write tid 4 to smithi19125844-5 2024-09-15T02:48:49.672 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 5 v 46 (ObjNum 68 snap 5 seq_num 68) dirty exists 2024-09-15T02:48:49.672 INFO:tasks.rados.rados.0.smithi191.stdout:117: left oid 5 (ObjNum 68 snap 5 seq_num 68) 2024-09-15T02:48:49.694 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:49.690+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:49.735 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:49.736 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:49.736 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:49.736 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:49.745 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:48:49.795 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:49.790+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:49.849 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:49.849 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:49.849 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:49.849 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:49.849 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:49.859 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:48:49.896 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:49.887+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:49.997 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:49.991+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:237 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:50.000 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4320787519999998, 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013334751000000001, 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 237 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565426+0000", 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2820000000000001e-06 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.002 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565442+0000", 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6096999999999999e-05 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565484+0000", 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2092999999999998e-05 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565500+0000", 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5976000000000001e-05 2024-09-15T02:48:50.003 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565647+0000", 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014730900000000001 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565664+0000", 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6405e-05 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.004 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569523+0000", 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003859627 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569647+0000", 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000123292 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578757+0000", 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0091106699999999995 2024-09-15T02:48:50.005 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:238 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4319784159999998, 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016905343999999999, 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 238 2024-09-15T02:48:50.006 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.007 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565524+0000", 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2619999999999999e-06 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565531+0000", 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3300000000000001e-06 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565555+0000", 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3272999999999999e-05 2024-09-15T02:48:50.008 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565567+0000", 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.205e-05 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565812+0000", 2024-09-15T02:48:50.009 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024487000000000001 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565822+0000", 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0356999999999999e-05 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050794789999999996 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571110+0000", 2024-09-15T02:48:50.010 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020819100000000001 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582428+0000", 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011318531999999999 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:239 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4319042660000001, 2024-09-15T02:48:50.011 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017215766, 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 239 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.012 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565598+0000", 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0669999999999999e-06 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565604+0000", 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7069999999999997e-06 2024-09-15T02:48:50.013 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565633+0000", 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9546000000000001e-05 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565642+0000", 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0950000000000002e-06 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565870+0000", 2024-09-15T02:48:50.014 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00022745799999999999 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565885+0000", 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.525e-05 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571578+0000", 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056929210000000001 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.015 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571650+0000", 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1894e-05 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582813+0000", 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011162828 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.016 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4267148269999996, 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001307866, 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.017 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.018 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570852+0000", 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5881000000000002e-05 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570868+0000", 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5452e-05 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.019 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572031+0000", 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011636719999999999 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572094+0000", 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2860999999999993e-05 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.020 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.9s0 39/33 ECSubRead(tid=26, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.426032534, 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00082345300000000003, 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.021 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571469+0000", 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.022 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571509+0000", 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0441999999999997e-05 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571520+0000", 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0655e-05 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572172+0000", 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065217599999999997 2024-09-15T02:48:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572292+0000", 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012018 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.9s0 39/33 ECSubRead(tid=27, to_read={3:94f52c52:::smithi19125844-20:head=0,81920,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.425582576, 2024-09-15T02:48:50.024 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00070841600000000002, 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.025 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571919+0000", 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572343+0000", 2024-09-15T02:48:50.026 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042437699999999998 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572351+0000", 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3049999999999993e-06 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572525+0000", 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017411299999999999 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.027 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.572627+0000", 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010162099999999999 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.578065+0000", 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4194365680000001, 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071418700000000002, 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.028 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578065+0000", 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578065+0000", 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.029 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578066+0000", 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4419999999999999e-06 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578232+0000", 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016620099999999999 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578274+0000", 2024-09-15T02:48:50.030 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2132999999999998e-05 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578281+0000", 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5200000000000003e-06 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578333+0000", 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1780999999999998e-05 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.031 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578779+0000", 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00044611 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.581704+0000", 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4157972819999998, 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00075500399999999998, 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.032 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581704+0000", 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581704+0000", 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.033 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581705+0000", 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.02e-06 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581782+0000", 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7280000000000005e-05 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581905+0000", 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000123103 2024-09-15T02:48:50.034 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581913+0000", 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0010000000000001e-06 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581993+0000", 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9396999999999999e-05 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582459+0000", 2024-09-15T02:48:50.035 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000466203 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 39/33 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.4156745060000002, 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00099855899999999999, 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.036 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581827+0000", 2024-09-15T02:48:50.037 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3399999999999999e-07 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.581992+0000", 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016515799999999999 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582026+0000", 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3154999999999997e-05 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.038 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582031+0000", 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4569999999999996e-06 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582494+0000", 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00046331600000000002 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582825+0000", 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00033083899999999998 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.039 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/33 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.607949+0000", 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.3895520189999999, 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0034879989999999999, 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607949+0000", 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607949+0000", 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607950+0000", 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.041 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.607987+0000", 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7363000000000001e-05 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608064+0000", 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6942999999999998e-05 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608075+0000", 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0793e-05 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.042 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608192+0000", 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0001169 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.608205+0000", 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3338e-05 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.611395+0000", 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0031900520000000001 2024-09-15T02:48:50.043 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.611437+0000", 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2175000000000002e-05 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.as4 44/33 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=44'15, trim_to=0'0, roll_forward_to=43'13))", 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.610419+0000", 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.3870817630000001, 2024-09-15T02:48:50.044 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0039176760000000001, 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610419+0000", 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610419+0000", 2024-09-15T02:48:50.045 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610420+0000", 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.04e-07 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610442+0000", 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2770000000000001e-05 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.046 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610490+0000", 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7515999999999997e-05 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610499+0000", 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6740000000000003e-06 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610562+0000", 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3485000000000006e-05 2024-09-15T02:48:50.047 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610575+0000", 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.306e-05 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614295+0000", 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0037195029999999999 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.048 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.614337+0000", 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2463999999999999e-05 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=37, last_complete=44'23, committed=1, applied=1))", 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.617252+0000", 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.3802496950000001, 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001360766, 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.049 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617252+0000", 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617252+0000", 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.050 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617253+0000", 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1429999999999999e-06 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617264+0000", 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0946e-05 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617334+0000", 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0339999999999994e-05 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617343+0000", 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7960000000000001e-06 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618588+0000", 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012452800000000001 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618612+0000", 2024-09-15T02:48:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4261e-05 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=37, last_complete=44'23, committed=1, applied=1))", 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.617298+0000", 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.3802030359999999, 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001404261, 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617298+0000", 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617298+0000", 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617299+0000", 2024-09-15T02:48:50.054 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.268e-06 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617307+0000", 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0590000000000003e-06 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617351+0000", 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3371999999999997e-05 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.055 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617360+0000", 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0890000000000006e-06 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618655+0000", 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012947939999999999 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618702+0000", 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7679000000000002e-05 2024-09-15T02:48:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/33 ECSubWrite(tid=31, reqid=osd.0.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.617345+0000", 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.3801558840000001, 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001075194, 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.057 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617345+0000", 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617345+0000", 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617346+0000", 2024-09-15T02:48:50.058 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.088e-06 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617366+0000", 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9707999999999998e-05 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617417+0000", 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1381000000000002e-05 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.059 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617425+0000", 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2960000000000001e-06 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617487+0000", 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.1768999999999998e-05 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617494+0000", 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4170000000000003e-06 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.060 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618379+0000", 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088493600000000001 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618421+0000", 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1598999999999999e-05 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.061 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=37, last_complete=44'23, committed=1, applied=1))", 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.617560+0000", 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.379941039, 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011359580000000001, 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617560+0000", 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.062 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617560+0000", 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617561+0000", 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1939999999999999e-06 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617568+0000", 2024-09-15T02:48:50.063 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0999999999999998e-06 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617636+0000", 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7599000000000001e-05 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617651+0000", 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5362999999999998e-05 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.064 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618687+0000", 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010357490000000001 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618696+0000", 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9530000000000005e-06 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.065 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=37, last_complete=44'23, committed=1, applied=1))", 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.617752+0000", 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.379749224, 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012459109999999999, 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617752+0000", 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.066 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617752+0000", 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617753+0000", 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2900000000000002e-07 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617762+0000", 2024-09-15T02:48:50.067 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7110000000000001e-06 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618898+0000", 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001136815 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618907+0000", 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8370000000000003e-06 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.068 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618980+0000", 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2929000000000006e-05 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618998+0000", 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7689999999999998e-05 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.as4 44/33 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:48:50.069 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618006+0000", 2024-09-15T02:48:50.070 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.379495352, 2024-09-15T02:48:50.070 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0023119780000000001, 2024-09-15T02:48:50.070 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.070 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.070 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.070 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618006+0000", 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618006+0000", 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.071 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618007+0000", 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618021+0000", 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4538e-05 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618086+0000", 2024-09-15T02:48:50.072 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.4428e-05 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618093+0000", 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4000000000000003e-06 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618167+0000", 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4010000000000005e-05 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.073 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618178+0000", 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1421e-05 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620282+0000", 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021034420000000001 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.620318+0000", 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5939000000000003e-05 2024-09-15T02:48:50.074 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.3786464, 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.075 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.076 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:48:50.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.fs2 45/33 ECSubRead(tid=32, to_read={3:fc5a0702:::smithi19125844-26:head=0,106496,0}, subchunks={3:fc5a0702:::smithi19125844-26:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:49.634412+0000", 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.36308918400000001, 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00082829599999999996, 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634412+0000", 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.079 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634412+0000", 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634413+0000", 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.13e-07 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634429+0000", 2024-09-15T02:48:50.080 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6362999999999999e-05 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634583+0000", 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015380799999999999 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634609+0000", 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6307000000000001e-05 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.081 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634762+0000", 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015328 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.635240+0000", 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00047792499999999998 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.fs2 45/33 ECSubRead(tid=34, to_read={3:fc5a0702:::smithi19125844-26:head=0,106496,0}, subchunks={3:fc5a0702:::smithi19125844-26:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.082 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:49.637058+0000", 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.36044342600000001, 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00098930599999999991, 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637058+0000", 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:50.083 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637058+0000", 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637059+0000", 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.683e-06 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637080+0000", 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0973000000000001e-05 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.084 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637204+0000", 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012401799999999999 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637223+0000", 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8960000000000001e-05 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.637348+0000", 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012472300000000001 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:50.085 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.638047+0000", 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00069894900000000003 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:50.086 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:50.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:48:50.133 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:50.129+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:50.199 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:50.193+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:50.199 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:50.200 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:50.200 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:50.200 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:50.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:48:50.301 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:50.299+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:50.311 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:50.312 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:50.312 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:50.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:50.312 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:50.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:48:50.402 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:50.397+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:50.424 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:50.424 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:50.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:50.425 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:50.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.425 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:50.425 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8521487480000003, 2024-09-15T02:48:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:48:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:50.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:48:50.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:48:50.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:48:50.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:48:50.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:48:50.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:48:50.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:48:50.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:48:50.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:48:50.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:48:50.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:48:50.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:48:50.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:48:50.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:48:50.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:48:50.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:48:50.437 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.438 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.438 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.438 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:50.438 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:48:50.439 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8422064420000002, 2024-09-15T02:48:50.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099046280000000004, 2024-09-15T02:48:50.439 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.439 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.439 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:48:50.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:48:50.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579386+0000", 2024-09-15T02:48:50.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.536e-06 2024-09-15T02:48:50.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579616+0000", 2024-09-15T02:48:50.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230366 2024-09-15T02:48:50.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579660+0000", 2024-09-15T02:48:50.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4489e-05 2024-09-15T02:48:50.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579668+0000", 2024-09-15T02:48:50.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4800000000000004e-06 2024-09-15T02:48:50.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579731+0000", 2024-09-15T02:48:50.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2733999999999999e-05 2024-09-15T02:48:50.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579742+0000", 2024-09-15T02:48:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-15T02:48:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589278+0000", 2024-09-15T02:48:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095358219999999994 2024-09-15T02:48:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589289+0000", 2024-09-15T02:48:50.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0596e-05 2024-09-15T02:48:50.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.453 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.454 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:50.454 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582826+0000", 2024-09-15T02:48:50.454 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8387645179999996, 2024-09-15T02:48:50.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064948000000000002, 2024-09-15T02:48:50.454 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.455 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.455 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582826+0000", 2024-09-15T02:48:50.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582826+0000", 2024-09-15T02:48:50.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582827+0000", 2024-09-15T02:48:50.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5300000000000003e-07 2024-09-15T02:48:50.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582848+0000", 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0890000000000002e-05 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582897+0000", 2024-09-15T02:48:50.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9184999999999997e-05 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582906+0000", 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2080000000000006e-06 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583564+0000", 2024-09-15T02:48:50.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065821499999999997 2024-09-15T02:48:50.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583574+0000", 2024-09-15T02:48:50.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9820000000000003e-06 2024-09-15T02:48:50.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589310+0000", 2024-09-15T02:48:50.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057356109999999998 2024-09-15T02:48:50.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589321+0000", 2024-09-15T02:48:50.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0856000000000001e-05 2024-09-15T02:48:50.465 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.465 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.465 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.466 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=97, reqid=client.4526.0:223, at_version=0'0, trim_to=0'0, roll_forward_to=39'12))", 2024-09-15T02:48:50.466 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.602279+0000", 2024-09-15T02:48:50.466 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8193113729999997, 2024-09-15T02:48:50.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011563400000000001, 2024-09-15T02:48:50.466 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.466 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602279+0000", 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602279+0000", 2024-09-15T02:48:50.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602281+0000", 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.502e-06 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602301+0000", 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0216000000000001e-05 2024-09-15T02:48:50.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602372+0000", 2024-09-15T02:48:50.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1353999999999995e-05 2024-09-15T02:48:50.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602381+0000", 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2930000000000006e-06 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602491+0000", 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000109768 2024-09-15T02:48:50.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602513+0000", 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.158e-05 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603347+0000", 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083392399999999999 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603435+0000", 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8702999999999994e-05 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.1s2 39/33 ECSubRead(tid=98, to_read={3:8610d5f9:::smithi19125844-1:head=0,4096,0}, subchunks={3:8610d5f9:::smithi19125844-1:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.602902+0000", 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8186882669999997, 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028495899999999999, 2024-09-15T02:48:50.472 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602902+0000", 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602902+0000", 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602903+0000", 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.02e-06 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602918+0000", 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.451e-05 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602970+0000", 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2024999999999997e-05 2024-09-15T02:48:50.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.602982+0000", 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2459e-05 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603057+0000", 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4828999999999998e-05 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.603187+0000", 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130116 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 40/33 ECSubRead(tid=18, to_read={3:517ce0e6:::smithi19125844-47:head=0,196608,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.556401+0000", 2024-09-15T02:48:50.476 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8651899480000003, 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00085077099999999995, 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556401+0000", 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556401+0000", 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556401+0000", 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556414+0000", 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3142000000000001e-05 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556513+0000", 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8708000000000004e-05 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556530+0000", 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7757e-05 2024-09-15T02:48:50.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.556625+0000", 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4933999999999996e-05 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.557251+0000", 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00062589000000000004 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 40/33 ECSubRead(tid=19, to_read={3:517ce0e6:::smithi19125844-47:head=0,196608,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.558499+0000", 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8630913180000004, 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041788300000000002, 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558499+0000", 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558499+0000", 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558501+0000", 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.615e-06 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558519+0000", 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8025000000000001e-05 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558632+0000", 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011349399999999999 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558649+0000", 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7122e-05 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558737+0000", 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7884000000000001e-05 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.558917+0000", 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000179743 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 40/33 ECSubRead(tid=20, to_read={3:517ce0e6:::smithi19125844-47:head=0,196608,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.560078+0000", 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8615125189999997, 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00050736200000000003, 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560078+0000", 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560078+0000", 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560079+0000", 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3969999999999999e-06 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560095+0000", 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5786000000000001e-05 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560229+0000", 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013436899999999999 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560253+0000", 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3980000000000001e-05 2024-09-15T02:48:50.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560402+0000", 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014817399999999999 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.560585+0000", 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000183656 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.489 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/33 ECSubWrite(tid=160, reqid=osd.11.0:160, at_version=44'19, trim_to=0'0, roll_forward_to=43'17))", 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.604074+0000", 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.817516637, 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029528079999999999, 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604074+0000", 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604074+0000", 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604075+0000", 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3e-07 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604176+0000", 2024-09-15T02:48:50.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101846 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604312+0000", 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000135926 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604333+0000", 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0523e-05 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604465+0000", 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132185 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.604500+0000", 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4759999999999999e-05 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606975+0000", 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024749519999999999 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607027+0000", 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1885999999999998e-05 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/33 ECSubWrite(tid=113, reqid=osd.13.0:113, at_version=44'14, trim_to=0'0, roll_forward_to=39'12))", 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.605748+0000", 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.8158426749999999, 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044098280000000002, 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.494 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605748+0000", 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605748+0000", 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605748+0000", 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9799999999999999e-07 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605792+0000", 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3844000000000002e-05 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605876+0000", 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4405000000000002e-05 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605900+0000", 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3159e-05 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605972+0000", 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2409000000000004e-05 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605986+0000", 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4296e-05 2024-09-15T02:48:50.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610119+0000", 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041326339999999996 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610158+0000", 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8683000000000002e-05 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/33 ECSubWrite(tid=161, reqid=osd.11.0:161, at_version=44'21, trim_to=0'0, roll_forward_to=43'17))", 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.607052+0000", 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.8145385190000001, 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045832210000000002, 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607052+0000", 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607052+0000", 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607053+0000", 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3699999999999999e-07 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607090+0000", 2024-09-15T02:48:50.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7243999999999998e-05 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607177+0000", 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6984999999999995e-05 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607187+0000", 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5850000000000006e-06 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607280+0000", 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3758999999999995e-05 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.607294+0000", 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3901e-05 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611601+0000", 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043067629999999999 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611635+0000", 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4047000000000001e-05 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.505 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 44/33 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=44'15, trim_to=0'0, roll_forward_to=43'13))", 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.610841+0000", 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.8107492949999999, 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013982879999999999, 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610841+0000", 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610841+0000", 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610842+0000", 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:48:50.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610879+0000", 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7537999999999999e-05 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610957+0000", 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8107000000000003e-05 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610970+0000", 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3016e-05 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611208+0000", 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000237321 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611221+0000", 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3644e-05 2024-09-15T02:48:50.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612213+0000", 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00099152099999999998 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612239+0000", 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6761e-05 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/33 ECSubWrite(tid=119, reqid=osd.13.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.612853+0000", 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.8087370270000001, 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034907200000000001, 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612853+0000", 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612853+0000", 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612855+0000", 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.59e-06 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612876+0000", 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.069e-05 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612953+0000", 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7075000000000003e-05 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612966+0000", 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3388e-05 2024-09-15T02:48:50.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613038+0000", 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1548e-05 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613052+0000", 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4226999999999999e-05 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.616265+0000", 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032130359999999998 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.616344+0000", 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9165999999999994e-05 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 44/33 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.618519+0000", 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.803070972, 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001460638, 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618519+0000", 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618519+0000", 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618521+0000", 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3e-06 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618537+0000", 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6268000000000001e-05 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618586+0000", 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9138000000000003e-05 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618600+0000", 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4234e-05 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618675+0000", 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4381000000000005e-05 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618695+0000", 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0131000000000001e-05 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619951+0000", 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012563559999999999 2024-09-15T02:48:50.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619980+0000", 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.883e-05 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/33 ECSubWrite(tid=162, reqid=osd.11.0:161, at_version=0'0, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.619446+0000", 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.802144024, 2024-09-15T02:48:50.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054007630000000003, 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619446+0000", 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619446+0000", 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619447+0000", 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3e-07 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619462+0000", 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5121e-05 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619503+0000", 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0584999999999999e-05 2024-09-15T02:48:50.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619510+0000", 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0890000000000002e-06 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619561+0000", 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1085000000000003e-05 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619572+0000", 2024-09-15T02:48:50.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0496000000000001e-05 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.624814+0000", 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005242643 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.624847+0000", 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3013999999999999e-05 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.524 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.525 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.525 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.525 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:50.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:48:50.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:48:50.527 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:50.522+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:50.616 INFO:tasks.rados.rados.0.smithi191.stdout:114: done (3 left) 2024-09-15T02:48:50.617 INFO:tasks.rados.rados.0.smithi191.stdout:117: done (2 left) 2024-09-15T02:48:50.617 INFO:tasks.rados.rados.0.smithi191.stdout:118: done (1 left) 2024-09-15T02:48:50.617 INFO:tasks.rados.rados.0.smithi191.stdout:120: done (0 left) 2024-09-15T02:48:50.617 INFO:tasks.rados.rados.0.smithi191.stdout:121: read oid 11 snap -1 2024-09-15T02:48:50.617 INFO:tasks.rados.rados.0.smithi191.stdout:121: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:48:50.617 INFO:tasks.rados.rados.0.smithi191.stdout:122: setattr oid 13 current snap is 6 2024-09-15T02:48:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:48:50.621 INFO:tasks.rados.rados.0.smithi191.stdout:123: copy_from oid 3 from oid 26 current snap is 6 2024-09-15T02:48:50.621 INFO:tasks.rados.rados.0.smithi191.stdout:124: read oid 5 snap 4 2024-09-15T02:48:50.621 INFO:tasks.rados.rados.0.smithi191.stdout:124: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-15T02:48:50.621 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 48 current snap is 6 2024-09-15T02:48:50.621 INFO:tasks.rados.rados.0.smithi191.stdout:125: seq_num 70 ranges {2785280=802816,3588096=376832} 2024-09-15T02:48:50.627 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:50.622+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:50.637 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:48:50.637 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:50.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:48:50.648 INFO:tasks.rados.rados.0.smithi191.stdout:125: writing smithi19125844-48 from 2785280 to 3588096 tid 1 2024-09-15T02:48:50.651 INFO:tasks.rados.rados.0.smithi191.stdout:125: writing smithi19125844-48 from 3588096 to 3964928 tid 2 2024-09-15T02:48:50.651 INFO:tasks.rados.rados.0.smithi191.stdout:126: rmattr oid 20 current snap is 6 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:126: done (5 left) 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:127: read oid 19 snap -1 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:127: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:128: rollback oid 41 current snap is 6 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 41 to 4 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:129: snap_create 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:123: finishing copy_from racing read to smithi19125844-3 2024-09-15T02:48:50.652 INFO:tasks.rados.rados.0.smithi191.stdout:124: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-15T02:48:50.686 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:50.686 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 13 v 10 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:123: finishing copy_from to smithi19125844-3 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 3 v 10 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:128: finishing rollback tid 0 to smithi19125844-41 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:128: finishing rollback tid 1 to smithi19125844-41 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:128: finishing rollback tid 2 to smithi19125844-41 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 41 v 50 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:125: finishing write tid 1 to smithi19125844-48 2024-09-15T02:48:50.687 INFO:tasks.rados.rados.0.smithi191.stdout:125: finishing write tid 2 to smithi19125844-48 2024-09-15T02:48:50.716 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:50.729 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:50.726+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:50.744 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:50.745 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:48:50.745 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:48:50.745 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:48:50.745 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:50.747 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:50.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:48:50.775 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:50.803 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:50.829 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:50.825+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:50.831 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:50.854 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 39/33 ECSubWrite(tid=91, reqid=client.4526.0:228, at_version=39'24, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 7.2820731289999996, 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048861590000000002, 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.855 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569082+0000", 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9400000000000005e-07 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569139+0000", 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7068999999999999e-05 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569277+0000", 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013777200000000001 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569299+0000", 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2033000000000001e-05 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569495+0000", 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000195336 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569523+0000", 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.879e-05 2024-09-15T02:48:50.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.573920+0000", 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043971050000000001 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.573968+0000", 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7360000000000001e-05 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 7.2814138269999997, 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:48:50.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 7.2688976419999998, 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.864 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:48:50.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 7.2678584820000003, 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:48:50.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:48:50.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.2734388089999999, 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081904660000000004, 2024-09-15T02:48:50.873 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.577716+0000", 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4799999999999999e-07 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578238+0000", 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052198700000000002 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578284+0000", 2024-09-15T02:48:50.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5556000000000002e-05 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2568000000000001e-05 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578387+0000", 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0867999999999998e-05 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.578406+0000", 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8995999999999999e-05 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.585856+0000", 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074492930000000001 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.585906+0000", 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0850000000000003e-05 2024-09-15T02:48:50.877 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 40/33 ECSubWrite(tid=100, reqid=client.4526.0:245, at_version=40'29, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:44.589294+0000", 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.2618603100000003, 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047200760000000001, 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589294+0000", 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589294+0000", 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589296+0000", 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1599999999999999e-06 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589437+0000", 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014099300000000001 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589488+0000", 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0937999999999999e-05 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589499+0000", 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1022000000000001e-05 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589569+0000", 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0623000000000005e-05 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.589582+0000", 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2649999999999999e-05 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.593939+0000", 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043576379999999996 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:44.594015+0000", 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5051999999999999e-05 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 41/33 ECSubWrite(tid=104, reqid=osd.13.0:104, at_version=41'32, trim_to=0'0, roll_forward_to=40'30))", 2024-09-15T02:48:50.882 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:45.601636+0000", 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.2495188830000004, 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058116670000000004, 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601636+0000", 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601636+0000", 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601636+0000", 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.44e-07 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601665+0000", 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8093999999999999e-05 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601704+0000", 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9694999999999999e-05 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601713+0000", 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0850000000000003e-06 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601769+0000", 2024-09-15T02:48:50.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6153999999999998e-05 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.601779+0000", 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8530000000000004e-06 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.607408+0000", 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005628325 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.607448+0000", 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9916999999999997e-05 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 44/33 ECSubWrite(tid=113, reqid=osd.13.0:113, at_version=44'14, trim_to=0'0, roll_forward_to=39'12))", 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.605175+0000", 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2459797720000001, 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050839040000000002, 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.887 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605175+0000", 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605175+0000", 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605175+0000", 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5400000000000002e-07 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605214+0000", 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8361000000000001e-05 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605530+0000", 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031631600000000003 2024-09-15T02:48:50.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605552+0000", 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1912e-05 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605794+0000", 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024216799999999999 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.605805+0000", 2024-09-15T02:48:50.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1094000000000001e-05 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610209+0000", 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044034340000000003 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610259+0000", 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0265000000000001e-05 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 44/33 ECSubWrite(tid=161, reqid=osd.11.0:161, at_version=44'21, trim_to=0'0, roll_forward_to=43'17))", 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.606482+0000", 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2446731309999999, 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006665023, 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606482+0000", 2024-09-15T02:48:50.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606482+0000", 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606483+0000", 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0499999999999999e-06 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606520+0000", 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7391999999999997e-05 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606583+0000", 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2711000000000003e-05 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606593+0000", 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9879999999999999e-06 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606661+0000", 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8608000000000004e-05 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.606678+0000", 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6183000000000001e-05 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613102+0000", 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064247280000000002 2024-09-15T02:48:50.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613147+0000", 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4363000000000002e-05 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.fs4 44/33 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.608467+0000", 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2426878029999999, 2024-09-15T02:48:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065039240000000003, 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608467+0000", 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608467+0000", 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608467+0000", 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7800000000000002e-07 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608498+0000", 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0443999999999999e-05 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608643+0000", 2024-09-15T02:48:50.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014490999999999999 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608656+0000", 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3692e-05 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608752+0000", 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5180999999999996e-05 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608765+0000", 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.329e-05 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.614956+0000", 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061910699999999999 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.614971+0000", 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4959e-05 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.900 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 44/33 ECSubWrite(tid=117, reqid=osd.13.0:117, at_version=44'40, trim_to=0'0, roll_forward_to=43'38))", 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.608700+0000", 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2424542789999999, 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062506020000000001, 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608700+0000", 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608700+0000", 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608701+0000", 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1200000000000001e-07 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608735+0000", 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3683999999999998e-05 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608800+0000", 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4488999999999999e-05 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608809+0000", 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2599999999999994e-06 2024-09-15T02:48:50.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608886+0000", 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7140999999999999e-05 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.608904+0000", 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.802e-05 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.614925+0000", 2024-09-15T02:48:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060211220000000003 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.614951+0000", 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5973999999999999e-05 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 44/33 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=44'42, trim_to=0'0, roll_forward_to=43'38))", 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.609574+0000", 2024-09-15T02:48:50.905 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2415808720000001, 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074590309999999996, 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.609574+0000", 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.609574+0000", 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.609575+0000", 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8200000000000008e-07 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.609621+0000", 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5989000000000003e-05 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.609676+0000", 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5000000000000002e-05 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.609688+0000", 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2323999999999999e-05 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612525+0000", 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028365930000000001 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612541+0000", 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6311e-05 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617012+0000", 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004470941 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617033+0000", 2024-09-15T02:48:50.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0891e-05 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as2 44/33 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=44'15, trim_to=0'0, roll_forward_to=43'13))", 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.610964+0000", 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2401910009999999, 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059889130000000002, 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610964+0000", 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610964+0000", 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610964+0000", 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0600000000000001e-07 2024-09-15T02:48:50.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.610994+0000", 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9920000000000002e-05 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611072+0000", 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7872000000000003e-05 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611082+0000", 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0387000000000001e-05 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611196+0000", 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011425399999999999 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.611220+0000", 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3774e-05 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.616915+0000", 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056945909999999997 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.616953+0000", 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7808999999999997e-05 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.914 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 44/33 ECSubWrite(tid=119, reqid=osd.13.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.612806+0000", 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.238348835, 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0040873610000000003, 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612806+0000", 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612806+0000", 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612807+0000", 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612825+0000", 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7368e-05 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612930+0000", 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010580499999999999 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612967+0000", 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6641999999999999e-05 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613073+0000", 2024-09-15T02:48:50.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105641 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613091+0000", 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7881999999999999e-05 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.616855+0000", 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037643210000000002 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.616893+0000", 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8392000000000001e-05 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as2 44/33 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.618591+0000", 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2325635679999998, 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060700149999999998, 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.919 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618591+0000", 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618591+0000", 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618592+0000", 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3200000000000004e-07 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618606+0000", 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4019000000000001e-05 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618655+0000", 2024-09-15T02:48:50.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8896000000000001e-05 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618662+0000", 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0310000000000001e-06 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618708+0000", 2024-09-15T02:48:50.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5998999999999998e-05 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618718+0000", 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8789999999999998e-06 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.624623+0000", 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059053179999999997 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.624661+0000", 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8141e-05 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 44/33 ECSubWrite(tid=162, reqid=osd.11.0:161, at_version=0'0, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.618986+0000", 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2321688160000002, 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057111319999999998, 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.924 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618986+0000", 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618986+0000", 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.618987+0000", 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9899999999999999e-07 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619000+0000", 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3261999999999999e-05 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619042+0000", 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1650999999999998e-05 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619049+0000", 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5630000000000003e-06 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619107+0000", 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7839e-05 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.619116+0000", 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:48:50.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.624667+0000", 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0055507280000000004 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.624697+0000", 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0009999999999999e-05 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 44/33 ECSubWrite(tid=120, reqid=osd.13.0:118, at_version=0'0, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.620894+0000", 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.2302603360000002, 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064706479999999999, 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.620894+0000", 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.620894+0000", 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.620895+0000", 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1800000000000005e-07 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.620907+0000", 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2111e-05 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.620951+0000", 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3773000000000003e-05 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.620959+0000", 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0099999999999995e-06 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.621016+0000", 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6502999999999998e-05 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.621036+0000", 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0784999999999998e-05 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.627337+0000", 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063002700000000002 2024-09-15T02:48:50.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.627365+0000", 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8377999999999998e-05 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.2172361140000001, 2024-09-15T02:48:50.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010839624000000001, 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.17e-07 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634471+0000", 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00055238900000000003 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634518+0000", 2024-09-15T02:48:50.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6437000000000003e-05 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634530+0000", 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1926999999999999e-05 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634612+0000", 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2633000000000007e-05 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634626+0000", 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4088e-05 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.644722+0000", 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010095615 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.644758+0000", 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418000000000003e-05 2024-09-15T02:48:50.937 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/33 ECSubWrite(tid=125, reqid=client.4526.0:295, at_version=45'45, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:49.645320+0000", 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.2058346339999999, 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041058600000000002, 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645320+0000", 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645320+0000", 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645321+0000", 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2190000000000001e-06 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645429+0000", 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107217 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645499+0000", 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0056999999999997e-05 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645516+0000", 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7348999999999999e-05 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645590+0000", 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4235999999999999e-05 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.645605+0000", 2024-09-15T02:48:50.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4888999999999999e-05 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.649381+0000", 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037754220000000001 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.649426+0000", 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5470999999999999e-05 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/33 ECSubWrite(tid=128, reqid=client.4526.0:296, at_version=0'0, trim_to=0'0, roll_forward_to=45'46))", 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:49.654981+0000", 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.1961739149999999, 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022858190000000001, 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.654981+0000", 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.654981+0000", 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.654982+0000", 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0200000000000001e-07 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:48:50.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.654994+0000", 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2261e-05 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.655030+0000", 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6276999999999998e-05 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.655037+0000", 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0650000000000001e-06 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.655082+0000", 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4512999999999999e-05 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.655088+0000", 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8279999999999999e-06 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.657235+0000", 2024-09-15T02:48:50.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021475140000000001 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.657267+0000", 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1559000000000001e-05 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:48:50.947 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:48:50.949 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:50.949 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:50.949 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:50.949 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:50.943+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:50.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:48:50.961 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:50.990 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:48:51.019 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:51.047 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:51.049 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:51.045+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:51.060 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.060 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:48:51.075 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:51.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:51.095+0000 7f6be4ddd700 0 -- 172.21.15.6:0/238733648 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f6be01920c0 msgr2=0x7f6be01944a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:48:51.112 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:51.142 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:51.145 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:51.145 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:51.150 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:51.143+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:51.154 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15] out_osds: [10] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:48:51.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:48:51.154 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-15T02:48:51.155 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 0.274101 2024-09-15T02:48:51.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 0.27410051235404875 2024-09-15T02:48:51.170 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.170 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:48:51.251 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:51.245+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:51.282 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.282 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:48:51.342 INFO:tasks.rados.rados.0.smithi191.stdout:127: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-15T02:48:51.352 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:51.343+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:51.387 INFO:tasks.rados.rados.0.smithi191.stdout:125: finishing write tid 3 to smithi19125844-48 2024-09-15T02:48:51.387 INFO:tasks.rados.rados.0.smithi191.stdout:125: finishing write tid 4 to smithi19125844-48 2024-09-15T02:48:51.387 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 25 (ObjNum 70 snap 6 seq_num 70) dirty exists 2024-09-15T02:48:51.387 INFO:tasks.rados.rados.0.smithi191.stdout:125: left oid 48 (ObjNum 70 snap 6 seq_num 70) 2024-09-15T02:48:51.390 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.390 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:48:51.439 INFO:tasks.rados.rados.0.smithi191.stdout:121: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:48:51.452 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:51.445+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:51.493 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.493 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:48:51.553 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:51.547+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:51.628 INFO:tasks.rados.rados.0.smithi191.stdout:121: done (7 left) 2024-09-15T02:48:51.628 INFO:tasks.rados.rados.0.smithi191.stdout:122: done (6 left) 2024-09-15T02:48:51.628 INFO:tasks.rados.rados.0.smithi191.stdout:123: done (5 left) 2024-09-15T02:48:51.628 INFO:tasks.rados.rados.0.smithi191.stdout:124: done (4 left) 2024-09-15T02:48:51.628 INFO:tasks.rados.rados.0.smithi191.stdout:125: done (3 left) 2024-09-15T02:48:51.629 INFO:tasks.rados.rados.0.smithi191.stdout:127: done (2 left) 2024-09-15T02:48:51.629 INFO:tasks.rados.rados.0.smithi191.stdout:128: done (1 left) 2024-09-15T02:48:51.629 INFO:tasks.rados.rados.0.smithi191.stdout:129: done (0 left) 2024-09-15T02:48:51.629 INFO:tasks.rados.rados.0.smithi191.stdout:130: read oid 20 snap -1 2024-09-15T02:48:51.629 INFO:tasks.rados.rados.0.smithi191.stdout:130: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:48:51.629 INFO:tasks.rados.rados.0.smithi191.stdout:131: delete oid 43 current snap is 7 2024-09-15T02:48:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.8 primary-affinity to 0.274101 (8179632) 2024-09-15T02:48:51.643 INFO:tasks.rados.rados.0.smithi191.stdout:130: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:48:51.651 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.651 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.654 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:51.647+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:51.660 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:48:51.755 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:51.753+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:51.766 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.766 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.775 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:48:51.856 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:51.853+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:51.877 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.877 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.886 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:48:51.957 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:51.954+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:51.985 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:51.985 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:51.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:48:52.058 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:52.051+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:52.105 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.105 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:48:52.158 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:52.151+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:52.214 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.214 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:48:52.260 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:52.257+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:52.309 INFO:tasks.rados.rados.0.smithi191.stdout:130: done (1 left) 2024-09-15T02:48:52.309 INFO:tasks.rados.rados.0.smithi191.stdout:131: done (0 left) 2024-09-15T02:48:52.309 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 37 current snap is 7 2024-09-15T02:48:52.309 INFO:tasks.rados.rados.0.smithi191.stdout:132: seq_num 72 ranges {1081344=442368,1523712=720896,2244608=49152} 2024-09-15T02:48:52.325 INFO:tasks.rados.rados.0.smithi191.stdout:132: writing smithi19125844-37 from 1081344 to 1523712 tid 1 2024-09-15T02:48:52.329 INFO:tasks.rados.rados.0.smithi191.stdout:132: writing smithi19125844-37 from 1523712 to 2244608 tid 2 2024-09-15T02:48:52.330 INFO:tasks.rados.rados.0.smithi191.stdout:132: writing smithi19125844-37 from 2244608 to 2293760 tid 3 2024-09-15T02:48:52.330 INFO:tasks.rados.rados.0.smithi191.stdout:133: rollback oid 15 current snap is 7 2024-09-15T02:48:52.330 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 15 to 5 2024-09-15T02:48:52.330 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 19 current snap is 7 2024-09-15T02:48:52.330 INFO:tasks.rados.rados.0.smithi191.stdout:134: seq_num 73 ranges {1212416=458752,1671168=475136,2146304=294912} 2024-09-15T02:48:52.339 INFO:tasks.rados.rados.0.smithi191.stdout:134: writing smithi19125844-19 from 1212416 to 1671168 tid 1 2024-09-15T02:48:52.343 INFO:tasks.rados.rados.0.smithi191.stdout:134: writing smithi19125844-19 from 1671168 to 2146304 tid 2 2024-09-15T02:48:52.345 INFO:tasks.rados.rados.0.smithi191.stdout:134: writing smithi19125844-19 from 2146304 to 2441216 tid 3 2024-09-15T02:48:52.345 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 36 current snap is 7 2024-09-15T02:48:52.345 INFO:tasks.rados.rados.0.smithi191.stdout:133: finishing rollback tid 0 to smithi19125844-15 2024-09-15T02:48:52.345 INFO:tasks.rados.rados.0.smithi191.stdout:135: seq_num 74 ranges {589824=475136,1064960=622592,1687552=147456} 2024-09-15T02:48:52.351 INFO:tasks.rados.rados.0.smithi191.stdout:135: writing smithi19125844-36 from 589824 to 1064960 tid 1 2024-09-15T02:48:52.357 INFO:tasks.rados.rados.0.smithi191.stdout:135: writing smithi19125844-36 from 1064960 to 1687552 tid 2 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:135: writing smithi19125844-36 from 1687552 to 1835008 tid 3 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:133: finishing rollback tid 1 to smithi19125844-15 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:133: finishing rollback tid 2 to smithi19125844-15 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 15 v 18 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:132: finishing write tid 1 to smithi19125844-37 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:133: done (3 left) 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:136: read oid 45 snap -1 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:136: expect deleted 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 25 current snap is 7 2024-09-15T02:48:52.358 INFO:tasks.rados.rados.0.smithi191.stdout:137: seq_num 75 ranges {409600=491520,901120=770048} 2024-09-15T02:48:52.360 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:52.355+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:52.361 INFO:tasks.rados.rados.0.smithi191.stdout:137: writing smithi19125844-25 from 409600 to 901120 tid 1 2024-09-15T02:48:52.366 INFO:tasks.rados.rados.0.smithi191.stdout:137: writing smithi19125844-25 from 901120 to 1671168 tid 2 2024-09-15T02:48:52.366 INFO:tasks.rados.rados.0.smithi191.stdout:138: snap_remove snap 6 2024-09-15T02:48:52.373 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.374 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:48:52.461 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:52.457+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:52.476 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.476 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:48:52.564 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:52.557+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:52.587 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.587 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:48:52.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:48:52.645 INFO:tasks.rados.rados.0.smithi191.stdout:138: done (5 left) 2024-09-15T02:48:52.645 INFO:tasks.rados.rados.0.smithi191.stdout:139: snap_create 2024-09-15T02:48:52.645 INFO:tasks.rados.rados.0.smithi191.stdout:134: finishing write tid 1 to smithi19125844-19 2024-09-15T02:48:52.646 INFO:tasks.rados.rados.0.smithi191.stdout:132: finishing write tid 2 to smithi19125844-37 2024-09-15T02:48:52.646 INFO:tasks.rados.rados.0.smithi191.stdout:132: finishing write tid 3 to smithi19125844-37 2024-09-15T02:48:52.646 INFO:tasks.rados.rados.0.smithi191.stdout:134: finishing write tid 2 to smithi19125844-19 2024-09-15T02:48:52.646 INFO:tasks.rados.rados.0.smithi191.stdout:134: finishing write tid 3 to smithi19125844-19 2024-09-15T02:48:52.646 INFO:tasks.rados.rados.0.smithi191.stdout:134: finishing write tid 4 to smithi19125844-19 2024-09-15T02:48:52.646 INFO:tasks.rados.rados.0.smithi191.stdout:134: finishing write tid 5 to smithi19125844-19 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 19 v 30 (ObjNum 73 snap 7 seq_num 73) dirty exists 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:134: left oid 19 (ObjNum 73 snap 7 seq_num 73) 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:137: finishing write tid 1 to smithi19125844-25 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:137: finishing write tid 2 to smithi19125844-25 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:137: finishing write tid 3 to smithi19125844-25 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:137: finishing write tid 4 to smithi19125844-25 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 25 v 14 (ObjNum 75 snap 7 seq_num 75) dirty exists 2024-09-15T02:48:52.647 INFO:tasks.rados.rados.0.smithi191.stdout:137: left oid 25 (ObjNum 75 snap 7 seq_num 75) 2024-09-15T02:48:52.663 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:52.655+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:52.676 INFO:tasks.rados.rados.0.smithi191.stdout:135: finishing write tid 1 to smithi19125844-36 2024-09-15T02:48:52.676 INFO:tasks.rados.rados.0.smithi191.stdout:135: finishing write tid 2 to smithi19125844-36 2024-09-15T02:48:52.677 INFO:tasks.rados.rados.0.smithi191.stdout:135: finishing write tid 3 to smithi19125844-36 2024-09-15T02:48:52.677 INFO:tasks.rados.rados.0.smithi191.stdout:135: finishing write tid 4 to smithi19125844-36 2024-09-15T02:48:52.679 INFO:tasks.rados.rados.0.smithi191.stdout:135: finishing write tid 5 to smithi19125844-36 2024-09-15T02:48:52.679 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 36 v 29 (ObjNum 74 snap 7 seq_num 74) dirty exists 2024-09-15T02:48:52.679 INFO:tasks.rados.rados.0.smithi191.stdout:135: left oid 36 (ObjNum 74 snap 7 seq_num 74) 2024-09-15T02:48:52.704 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.704 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:48:52.764 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:52.761+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:52.813 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.814 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:48:52.868 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:52.863+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:52.923 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:52.923 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:52.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:48:52.966 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:52.959+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:53.034 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:53.034 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:53.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:48:53.067 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:53.061+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:53.148 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:53.148 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:53.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:48:53.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:48:53.168 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:53.166+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:53.180 INFO:tasks.rados.rados.0.smithi191.stdout:132: finishing write tid 4 to smithi19125844-37 2024-09-15T02:48:53.184 INFO:tasks.rados.rados.0.smithi191.stdout:132: finishing write tid 5 to smithi19125844-37 2024-09-15T02:48:53.184 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 37 v 21 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-15T02:48:53.184 INFO:tasks.rados.rados.0.smithi191.stdout:132: left oid 37 (ObjNum 72 snap 7 seq_num 72) 2024-09-15T02:48:53.259 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:53.259 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:53.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:48:53.269 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:53.261+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:53.295 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:53.324 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:53.354 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:53.361 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:53.361 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:53.370 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:53.365+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:53.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:48:53.384 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:53.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:53.440 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:53.470 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:53.470 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:53.463+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:53.480 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:53.480 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:53.480 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:48:53.480 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:53.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:48:53.499 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:53.526 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:53.555 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:53.572 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:53.565+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:53.584 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:48:53.588 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:53.589 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:53.589 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:48:53.589 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:48:53.589 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:53.598 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:48:53.614 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:53.642 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:132: done (5 left) 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:134: done (4 left) 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:135: done (3 left) 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:136: done (2 left) 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:137: done (1 left) 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:139: done (0 left) 2024-09-15T02:48:53.655 INFO:tasks.rados.rados.0.smithi191.stdout:140: setattr oid 48 current snap is 8 2024-09-15T02:48:53.661 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:48:53.661 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 2 current snap is 8 2024-09-15T02:48:53.661 INFO:tasks.rados.rados.0.smithi191.stdout:141: seq_num 76 ranges {737280=507904,1245184=507904,1753088=262144} 2024-09-15T02:48:53.672 INFO:tasks.rados.rados.0.smithi191.stdout:141: writing smithi19125844-2 from 737280 to 1245184 tid 1 2024-09-15T02:48:53.672 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:53.665+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:53.675 INFO:tasks.rados.rados.0.smithi191.stdout:141: writing smithi19125844-2 from 1245184 to 1753088 tid 2 2024-09-15T02:48:53.677 INFO:tasks.rados.rados.0.smithi191.stdout:141: writing smithi19125844-2 from 1753088 to 2015232 tid 3 2024-09-15T02:48:53.677 INFO:tasks.rados.rados.0.smithi191.stdout:142: copy_from oid 28 from oid 42 current snap is 8 2024-09-15T02:48:53.677 INFO:tasks.rados.rados.0.smithi191.stdout:143: setattr oid 10 current snap is 8 2024-09-15T02:48:53.677 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:53.678 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 27 (ObjNum 70 snap 6 seq_num 70) dirty exists 2024-09-15T02:48:53.681 INFO:tasks.rados.rados.0.smithi191.stdout:140: done (3 left) 2024-09-15T02:48:53.681 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 39 current snap is 8 2024-09-15T02:48:53.681 INFO:tasks.rados.rados.0.smithi191.stdout:144: seq_num 77 ranges {638976=524288,1163264=524288,1687552=245760} 2024-09-15T02:48:53.688 INFO:tasks.rados.rados.0.smithi191.stdout:144: writing smithi19125844-39 from 638976 to 1163264 tid 1 2024-09-15T02:48:53.691 INFO:tasks.rados.rados.0.smithi191.stdout:144: writing smithi19125844-39 from 1163264 to 1687552 tid 2 2024-09-15T02:48:53.693 INFO:tasks.rados.rados.0.smithi191.stdout:144: writing smithi19125844-39 from 1687552 to 1933312 tid 3 2024-09-15T02:48:53.693 INFO:tasks.rados.rados.0.smithi191.stdout:145: read oid 23 snap 5 2024-09-15T02:48:53.694 INFO:tasks.rados.rados.0.smithi191.stdout:145: expect deleted 2024-09-15T02:48:53.694 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 44 current snap is 8 2024-09-15T02:48:53.694 INFO:tasks.rados.rados.0.smithi191.stdout:146: seq_num 78 ranges {737280=557056,1294336=671744,1966080=98304} 2024-09-15T02:48:53.701 INFO:tasks.rados.rados.0.smithi191.stdout:146: writing smithi19125844-44 from 737280 to 1294336 tid 1 2024-09-15T02:48:53.706 INFO:tasks.rados.rados.0.smithi191.stdout:146: writing smithi19125844-44 from 1294336 to 1966080 tid 2 2024-09-15T02:48:53.706 INFO:tasks.rados.rados.0.smithi191.stdout:146: writing smithi19125844-44 from 1966080 to 2064384 tid 3 2024-09-15T02:48:53.706 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 10 v 25 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T02:48:53.706 INFO:tasks.rados.rados.0.smithi191.stdout:141: finishing write tid 1 to smithi19125844-2 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:141: finishing write tid 2 to smithi19125844-2 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:141: finishing write tid 3 to smithi19125844-2 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:141: finishing write tid 4 to smithi19125844-2 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:142: finishing copy_from racing read to smithi19125844-28 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:143: done (5 left) 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:145: done (4 left) 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:147: read oid 21 snap -1 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:147: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 1 current snap is 8 2024-09-15T02:48:53.707 INFO:tasks.rados.rados.0.smithi191.stdout:148: seq_num 79 ranges {1081344=573440,1654784=425984,2080768=344064} 2024-09-15T02:48:53.713 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:53.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:53.707+0000 7fb867fff700 0 -- 172.21.15.6:0/2986386148 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fb868194090 msgr2=0x7fb868196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:48:53.716 INFO:tasks.rados.rados.0.smithi191.stdout:148: writing smithi19125844-1 from 1081344 to 1654784 tid 1 2024-09-15T02:48:53.717 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:53.717 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:48:53.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:48:53.717 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 39/33 ECSubWrite(tid=91, reqid=client.4526.0:228, at_version=39'24, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.569123+0000", 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.146098314, 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027613859999999998, 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569123+0000", 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569123+0000", 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569124+0000", 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0550000000000001e-06 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569171+0000", 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7444000000000002e-05 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569236+0000", 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4961000000000004e-05 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569251+0000", 2024-09-15T02:48:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4613000000000001e-05 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569354+0000", 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102953 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569373+0000", 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9690999999999998e-05 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571820+0000", 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024464059999999999 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571884+0000", 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4263000000000005e-05 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 39/33 ECSubWrite(tid=16, reqid=client.4526.0:231, at_version=39'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.572486+0000", 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.142735076999999, 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022364559999999999, 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572486+0000", 2024-09-15T02:48:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572486+0000", 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572486+0000", 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8700000000000002e-07 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572524+0000", 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305999999999998e-05 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572611+0000", 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7034999999999996e-05 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572621+0000", 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572698+0000", 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7597999999999998e-05 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572713+0000", 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5074e-05 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574688+0000", 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001974876 2024-09-15T02:48:53.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574722+0000", 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4366999999999998e-05 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 39/33 ECSubWrite(tid=21, reqid=client.4526.0:234, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.573763+0000", 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.141457848, 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033820339999999999, 2024-09-15T02:48:53.727 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573763+0000", 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573763+0000", 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573764+0000", 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7800000000000002e-07 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573808+0000", 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3894999999999999e-05 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573885+0000", 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7587000000000001e-05 2024-09-15T02:48:53.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573897+0000", 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2082000000000001e-05 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574790+0000", 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089305600000000004 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574802+0000", 2024-09-15T02:48:53.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1647e-05 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577101+0000", 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022985779999999999 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577145+0000", 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4710999999999999e-05 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 39/33 ECSubWrite(tid=22, reqid=client.4526.0:235, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.574745+0000", 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.140475729, 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029092269999999999, 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574745+0000", 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574745+0000", 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574746+0000", 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4600000000000004e-07 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574794+0000", 2024-09-15T02:48:53.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8396000000000002e-05 2024-09-15T02:48:53.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574852+0000", 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7107000000000001e-05 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574859+0000", 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2169999999999997e-06 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575335+0000", 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047645100000000002 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575345+0000", 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.825e-06 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577632+0000", 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002287463 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577655+0000", 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2121999999999999e-05 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.737 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 39/33 ECSubWrite(tid=18, reqid=client.4526.0:233, at_version=39'11, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.575131+0000", 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.140089959999999, 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025011149999999999, 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575131+0000", 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575131+0000", 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575132+0000", 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.85e-07 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575152+0000", 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9930999999999999e-05 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575198+0000", 2024-09-15T02:48:53.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6350999999999997e-05 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575206+0000", 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4220000000000003e-06 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575277+0000", 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0969000000000004e-05 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575287+0000", 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0444e-05 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577604+0000", 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023169670000000001 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577632+0000", 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8246e-05 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 39/33 ECSubWrite(tid=23, reqid=client.4526.0:236, at_version=39'18, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.575181+0000", 2024-09-15T02:48:53.743 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.140040537999999, 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024869279999999998, 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575181+0000", 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575181+0000", 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575181+0000", 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8400000000000004e-07 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575206+0000", 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4766999999999999e-05 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575605+0000", 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039878299999999999 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575614+0000", 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5829999999999996e-06 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575955+0000", 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034037100000000001 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575966+0000", 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1574e-05 2024-09-15T02:48:53.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577658+0000", 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001692102 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.577667+0000", 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.064e-06 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.748 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 39/33 ECSubWrite(tid=19, reqid=client.4526.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.057379+0000", 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.6578424100000007, 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002112746, 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057379+0000", 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057379+0000", 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057380+0000", 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.776e-06 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057405+0000", 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4284e-05 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057484+0000", 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9684000000000005e-05 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057497+0000", 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3063e-05 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057678+0000", 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018086299999999999 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057716+0000", 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7783000000000001e-05 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.059434+0000", 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017180979999999999 2024-09-15T02:48:53.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.059491+0000", 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7194999999999997e-05 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:53.754 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1373968590000008, 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010162803999999999, 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577825+0000", 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9299999999999996e-07 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578183+0000", 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035738600000000001 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578258+0000", 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5891999999999998e-05 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578278+0000", 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9641999999999999e-05 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578517+0000", 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000238619 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578582+0000", 2024-09-15T02:48:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5091000000000004e-05 2024-09-15T02:48:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.587957+0000", 2024-09-15T02:48:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0093750499999999994 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.587987+0000", 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0230999999999999e-05 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 40/33 ECSubWrite(tid=100, reqid=client.4526.0:245, at_version=40'29, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.589678+0000", 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1255432790000004, 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003348373, 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589678+0000", 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589678+0000", 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589679+0000", 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0890000000000001e-06 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589771+0000", 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2373000000000005e-05 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589842+0000", 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589862+0000", 2024-09-15T02:48:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9505e-05 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589986+0000", 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124738 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589998+0000", 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1929999999999999e-05 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592992+0000", 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029937100000000001 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.593026+0000", 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.417e-05 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 44/33 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=44'21, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.597878+0000", 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.1173426080000004, 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050384940000000001, 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.597878+0000", 2024-09-15T02:48:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.597878+0000", 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.597879+0000", 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.46e-07 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.597924+0000", 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5046000000000002e-05 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598046+0000", 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121661 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598070+0000", 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4125999999999999e-05 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598251+0000", 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000181256 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598281+0000", 2024-09-15T02:48:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0448999999999999e-05 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602871+0000", 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045900150000000002 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602917+0000", 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5595e-05 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.600031+0000", 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.115190449, 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053886359999999996, 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600031+0000", 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600031+0000", 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600032+0000", 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600107+0000", 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5321999999999995e-05 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600211+0000", 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104021 2024-09-15T02:48:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600228+0000", 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6616000000000002e-05 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600583+0000", 2024-09-15T02:48:53.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035511599999999999 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600613+0000", 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9686e-05 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.605391+0000", 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047780369999999997 2024-09-15T02:48:53.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.605419+0000", 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 44/33 ECSubWrite(tid=117, reqid=osd.13.0:117, at_version=44'40, trim_to=0'0, roll_forward_to=43'38))", 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.608824+0000", 2024-09-15T02:48:53.776 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.1063975060000004, 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031452469999999999, 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.608824+0000", 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.608824+0000", 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.608824+0000", 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.28e-07 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.608910+0000", 2024-09-15T02:48:53.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5915000000000007e-05 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.608995+0000", 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5333000000000004e-05 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609003+0000", 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8310000000000001e-06 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609251+0000", 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024817399999999999 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609262+0000", 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0467e-05 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.611943+0000", 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026809709999999999 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.611969+0000", 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6327999999999999e-05 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/33 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.609245+0000", 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.10597659, 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00302316, 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609245+0000", 2024-09-15T02:48:53.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609245+0000", 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609245+0000", 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6800000000000001e-07 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609294+0000", 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9444000000000003e-05 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609464+0000", 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016941999999999999 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609473+0000", 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1949999999999992e-06 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609642+0000", 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016916899999999999 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609652+0000", 2024-09-15T02:48:53.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999993e-06 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.612242+0000", 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025903900000000001 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.612268+0000", 2024-09-15T02:48:53.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5724e-05 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 44/33 ECSubWrite(tid=31, reqid=osd.8.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.609319+0000", 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.1059023039999998, 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029368039999999999, 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.787 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609319+0000", 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609319+0000", 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609319+0000", 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8899999999999999e-07 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609331+0000", 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.113e-05 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609496+0000", 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000165246 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609503+0000", 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7429999999999996e-06 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609644+0000", 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014109499999999999 2024-09-15T02:48:53.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609653+0000", 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9120000000000004e-06 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.612205+0000", 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025525000000000001 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.612256+0000", 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0488999999999997e-05 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/33 ECSubWrite(tid=39, reqid=osd.14.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=44'25))", 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.346161+0000", 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.36905973, 2024-09-15T02:48:53.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026534050000000002, 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346161+0000", 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346161+0000", 2024-09-15T02:48:53.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346163+0000", 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.827e-06 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346193+0000", 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9796000000000001e-05 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346298+0000", 2024-09-15T02:48:53.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010475399999999999 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346316+0000", 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8147000000000001e-05 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346483+0000", 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016673299999999999 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346515+0000", 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2255000000000002e-05 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.348737+0000", 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022224879999999999 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.348815+0000", 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7404999999999995e-05 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0810873599999997, 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016823131000000002, 2024-09-15T02:48:53.799 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634136+0000", 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7940000000000001e-06 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634475+0000", 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033968099999999999 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634663+0000", 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018762999999999999 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634681+0000", 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7974000000000001e-05 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634905+0000", 2024-09-15T02:48:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022468600000000001 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634923+0000", 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7254000000000001e-05 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650923+0000", 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015999777999999999 2024-09-15T02:48:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650957+0000", 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4334e-05 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/33 ECSubWrite(tid=125, reqid=client.4526.0:295, at_version=45'45, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:53.804 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.645691+0000", 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0695297090000002, 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060612399999999999, 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645691+0000", 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645691+0000", 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645693+0000", 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8929999999999999e-06 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645864+0000", 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170376 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645985+0000", 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121002 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.646005+0000", 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0015000000000001e-05 2024-09-15T02:48:53.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647020+0000", 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001015127 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647037+0000", 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6748e-05 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651728+0000", 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046918380000000003 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651753+0000", 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4241000000000001e-05 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 46/33 ECSubWrite(tid=131, reqid=client.4526.0:306, at_version=46'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.629048+0000", 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.0861726699999998, 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002997918, 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629048+0000", 2024-09-15T02:48:53.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629048+0000", 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629050+0000", 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.31e-06 2024-09-15T02:48:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629473+0000", 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042335499999999999 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629883+0000", 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040994700000000002 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629901+0000", 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7473e-05 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629974+0000", 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3501e-05 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.629983+0000", 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7949999999999996e-06 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631977+0000", 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019937539999999999 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632046+0000", 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9783000000000006e-05 2024-09-15T02:48:53.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.0837792080000002, 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082044130000000007, 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.27e-07 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631889+0000", 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044670300000000001 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631941+0000", 2024-09-15T02:48:53.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1823999999999997e-05 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631956+0000", 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5143999999999999e-05 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632032+0000", 2024-09-15T02:48:53.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6149999999999994e-05 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632044+0000", 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2282e-05 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.639605+0000", 2024-09-15T02:48:53.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075610620000000003 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.639646+0000", 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0821000000000001e-05 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.820 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 46/33 ECSubWrite(tid=135, reqid=client.4526.0:319, at_version=46'48, trim_to=0'0, roll_forward_to=45'46))", 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.653117+0000", 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.062104454, 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020661550000000001, 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653117+0000", 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653117+0000", 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653118+0000", 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1659999999999999e-06 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653168+0000", 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9759999999999998e-05 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653237+0000", 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.957e-05 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653249+0000", 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2092999999999999e-05 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653327+0000", 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8102000000000005e-05 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.653342+0000", 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4538999999999999e-05 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.655148+0000", 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018065679999999999 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.655183+0000", 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4357000000000003e-05 2024-09-15T02:48:53.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:53.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:53.826 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:53.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:48:53.830 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:53.821+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:53.832 INFO:tasks.rados.rados.0.smithi191.stdout:148: writing smithi19125844-1 from 1654784 to 2080768 tid 2 2024-09-15T02:48:53.832 INFO:tasks.rados.rados.0.smithi191.stdout:148: writing smithi19125844-1 from 2080768 to 2424832 tid 3 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 46 current snap is 8 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:141: finishing write tid 5 to smithi19125844-2 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 2 v 21 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:141: left oid 2 (ObjNum 76 snap 8 seq_num 76) 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:144: finishing write tid 1 to smithi19125844-39 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:142: finishing copy_from to smithi19125844-28 2024-09-15T02:48:53.833 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 4 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:147: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:144: finishing write tid 2 to smithi19125844-39 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:144: finishing write tid 3 to smithi19125844-39 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:144: finishing write tid 4 to smithi19125844-39 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:144: finishing write tid 5 to smithi19125844-39 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 39 v 25 (ObjNum 77 snap 8 seq_num 77) dirty exists 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:144: left oid 39 (ObjNum 77 snap 8 seq_num 77) 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:146: finishing write tid 1 to smithi19125844-44 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:149: seq_num 80 ranges {770048=589824,1359872=573440,1933312=196608} 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:149: writing smithi19125844-46 from 770048 to 1359872 tid 1 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:149: writing smithi19125844-46 from 1359872 to 1933312 tid 2 2024-09-15T02:48:53.834 INFO:tasks.rados.rados.0.smithi191.stdout:149: writing smithi19125844-46 from 1933312 to 2129920 tid 3 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:141: done (6 left) 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:142: done (5 left) 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:144: done (4 left) 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:147: done (3 left) 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:150: copy_from oid 8 from oid 12 current snap is 8 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:146: finishing write tid 2 to smithi19125844-44 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:151: read oid 13 snap -1 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:151: expect (ObjNum 58 snap 0 seq_num 58) 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:152: read oid 35 snap -1 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:152: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-15T02:48:53.835 INFO:tasks.rados.rados.0.smithi191.stdout:148: finishing write tid 1 to smithi19125844-1 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:153: read oid 22 snap -1 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:153: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:154: read oid 39 snap 7 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:154: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 45 current snap is 8 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:148: finishing write tid 2 to smithi19125844-1 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:155: seq_num 81 ranges {0=606208,606208=720896,1327104=49152} 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:155: writing smithi19125844-45 from 0 to 606208 tid 1 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:155: writing smithi19125844-45 from 606208 to 1327104 tid 2 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:155: writing smithi19125844-45 from 1327104 to 1376256 tid 3 2024-09-15T02:48:53.836 INFO:tasks.rados.rados.0.smithi191.stdout:156: snap_create 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:148: finishing write tid 3 to smithi19125844-1 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:148: finishing write tid 4 to smithi19125844-1 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:148: finishing write tid 5 to smithi19125844-1 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 1 v 30 (ObjNum 79 snap 8 seq_num 79) dirty exists 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:148: left oid 1 (ObjNum 79 snap 8 seq_num 79) 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:153: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:152: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:151: expect (ObjNum 58 snap 0 seq_num 58) 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:149: finishing write tid 1 to smithi19125844-46 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:149: finishing write tid 2 to smithi19125844-46 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:149: finishing write tid 3 to smithi19125844-46 2024-09-15T02:48:53.837 INFO:tasks.rados.rados.0.smithi191.stdout:149: finishing write tid 4 to smithi19125844-46 2024-09-15T02:48:53.928 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:48:53.926+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:48:53.932 INFO:tasks.rados.rados.0.smithi191.stdout:146: finishing write tid 3 to smithi19125844-44 2024-09-15T02:48:53.932 INFO:tasks.rados.rados.0.smithi191.stdout:146: finishing write tid 4 to smithi19125844-44 2024-09-15T02:48:53.940 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:53.940 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:53.940 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:48:53.940 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:53.942 INFO:tasks.rados.rados.0.smithi191.stdout:146: finishing write tid 5 to smithi19125844-44 2024-09-15T02:48:53.942 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 44 v 21 (ObjNum 78 snap 8 seq_num 78) dirty exists 2024-09-15T02:48:53.942 INFO:tasks.rados.rados.0.smithi191.stdout:146: left oid 44 (ObjNum 78 snap 8 seq_num 78) 2024-09-15T02:48:53.947 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:53.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:48:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:48:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:53.953 INFO:tasks.rados.rados.0.smithi191.stdout:155: finishing write tid 1 to smithi19125844-45 2024-09-15T02:48:53.954 INFO:tasks.rados.rados.0.smithi191.stdout:155: finishing write tid 2 to smithi19125844-45 2024-09-15T02:48:53.954 INFO:tasks.rados.rados.0.smithi191.stdout:155: finishing write tid 3 to smithi19125844-45 2024-09-15T02:48:53.955 INFO:tasks.rados.rados.0.smithi191.stdout:155: finishing write tid 4 to smithi19125844-45 2024-09-15T02:48:53.958 INFO:tasks.rados.rados.0.smithi191.stdout:150: finishing copy_from racing read to smithi19125844-8 2024-09-15T02:48:53.958 INFO:tasks.rados.rados.0.smithi191.stdout:149: finishing write tid 5 to smithi19125844-46 2024-09-15T02:48:53.959 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 46 v 55 (ObjNum 80 snap 8 seq_num 80) dirty exists 2024-09-15T02:48:53.959 INFO:tasks.rados.rados.0.smithi191.stdout:149: left oid 46 (ObjNum 80 snap 8 seq_num 80) 2024-09-15T02:48:53.959 INFO:tasks.rados.rados.0.smithi191.stdout:155: finishing write tid 5 to smithi19125844-45 2024-09-15T02:48:53.959 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 45 v 29 (ObjNum 81 snap 8 seq_num 81) dirty exists 2024-09-15T02:48:53.959 INFO:tasks.rados.rados.0.smithi191.stdout:155: left oid 45 (ObjNum 81 snap 8 seq_num 81) 2024-09-15T02:48:54.030 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:54.025+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:54.038 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.038 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:54.038 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:48:54.039 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:48:54.039 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:48:54.130 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:54.127+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:54.146 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 39/33 ECSubWrite(tid=27, reqid=client.4526.0:220, at_version=39'14, trim_to=0'0, roll_forward_to=36'13))", 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.561443+0000", 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.581995254000001, 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027835149999999999, 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561443+0000", 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561443+0000", 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561443+0000", 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:48:54.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561690+0000", 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000246588 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561781+0000", 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1657999999999996e-05 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561800+0000", 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8649999999999999e-05 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561954+0000", 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015347200000000001 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.561981+0000", 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7514000000000001e-05 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.564117+0000", 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021354149999999999 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.564226+0000", 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109518 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.151 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 39/33 ECSubWrite(tid=28, reqid=client.4526.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=39'14))", 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.565499+0000", 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.577939244, 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023589610000000001, 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565499+0000", 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565499+0000", 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565500+0000", 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.683e-06 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565532+0000", 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1699999999999998e-05 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565627+0000", 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5418e-05 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565646+0000", 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8839e-05 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565948+0000", 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030192400000000002 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.565972+0000", 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3998999999999998e-05 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567813+0000", 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018406259999999999 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567858+0000", 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4771999999999998e-05 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 39/33 ECSubWrite(tid=91, reqid=client.4526.0:228, at_version=39'24, trim_to=0'0, roll_forward_to=36'22))", 2024-09-15T02:48:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.569135+0000", 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.574302911, 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021510029999999999, 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569135+0000", 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569135+0000", 2024-09-15T02:48:54.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569135+0000", 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5999999999999999e-07 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569172+0000", 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6202000000000003e-05 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569286+0000", 2024-09-15T02:48:54.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114047 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569298+0000", 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2238e-05 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569415+0000", 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116817 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.569432+0000", 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7621999999999998e-05 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571229+0000", 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017963499999999999 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.571286+0000", 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7266999999999999e-05 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 39/33 ECSubWrite(tid=16, reqid=client.4526.0:231, at_version=39'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.572956+0000", 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.570482290999999, 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021337050000000001, 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572956+0000", 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572956+0000", 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572956+0000", 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3599999999999999e-07 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.572997+0000", 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0982999999999997e-05 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573078+0000", 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0632999999999999e-05 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573089+0000", 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1839e-05 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573170+0000", 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0197000000000005e-05 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573188+0000", 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8181000000000001e-05 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575054+0000", 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018662710000000001 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575089+0000", 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5165e-05 2024-09-15T02:48:54.165 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 39/33 ECSubWrite(tid=19, reqid=client.4526.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.057418+0000", 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.086019981, 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022245110000000002, 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057418+0000", 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057418+0000", 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057420+0000", 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1770000000000001e-06 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057444+0000", 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4043e-05 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057531+0000", 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6755000000000005e-05 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057576+0000", 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4712000000000002e-05 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057707+0000", 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013184700000000001 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057741+0000", 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3346000000000002e-05 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.059570+0000", 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018291589999999999 2024-09-15T02:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.059642+0000", 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2471999999999993e-05 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.5654942409999997, 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014940116999999999, 2024-09-15T02:48:54.170 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577945+0000", 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.037e-06 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035186899999999999 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578459+0000", 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016206200000000001 2024-09-15T02:48:54.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578482+0000", 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3070000000000001e-05 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578643+0000", 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016161999999999999 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578670+0000", 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592839+0000", 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014168557 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592884+0000", 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4786000000000001e-05 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 40/33 ECSubWrite(tid=100, reqid=client.4526.0:245, at_version=40'29, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.589860+0000", 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.5535778699999998, 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040648510000000004, 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589860+0000", 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589860+0000", 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589861+0000", 2024-09-15T02:48:54.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.176e-06 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.589958+0000", 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6811000000000005e-05 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.590014+0000", 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6144999999999999e-05 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.590021+0000", 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2409999999999998e-06 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.590115+0000", 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4134000000000004e-05 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.590127+0000", 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1939e-05 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.593876+0000", 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037486289999999999 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.593925+0000", 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8776000000000002e-05 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/33 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=44'21, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:54.179 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.545415201, 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044055149999999996, 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:48:54.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.96e-07 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598066+0000", 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2595999999999997e-05 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598773+0000", 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070733500000000002 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598791+0000", 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8450000000000001e-05 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598890+0000", 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8510000000000004e-05 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598910+0000", 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0514999999999999e-05 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602080+0000", 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031698780000000001 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602428+0000", 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034783500000000001 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.183 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.600217+0000", 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.5432212459999999, 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057404500000000002, 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600217+0000", 2024-09-15T02:48:54.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600217+0000", 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600218+0000", 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.356e-06 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600491+0000", 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027357800000000002 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600579+0000", 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7216e-05 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600592+0000", 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2798e-05 2024-09-15T02:48:54.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600866+0000", 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027486999999999998 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600893+0000", 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6843e-05 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.605899+0000", 2024-09-15T02:48:54.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050058239999999999 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.605957+0000", 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7964999999999998e-05 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:48:54.188 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.5092321059999998, 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016865508000000001, 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:48:54.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0999999999999999e-07 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634501+0000", 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029489499999999999 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634605+0000", 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103488 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634624+0000", 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9530000000000001e-05 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634715+0000", 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1284000000000002e-05 2024-09-15T02:48:54.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634741+0000", 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5350999999999998e-05 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651036+0000", 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016295245999999999 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651071+0000", 2024-09-15T02:48:54.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5203999999999997e-05 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 45/33 ECSubWrite(tid=33, reqid=client.4526.0:301, at_version=45'16, trim_to=0'0, roll_forward_to=39'14))", 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.636411+0000", 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.5070268049999997, 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040876009999999997, 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636411+0000", 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636411+0000", 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636411+0000", 2024-09-15T02:48:54.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1199999999999999e-07 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636677+0000", 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026585600000000002 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636765+0000", 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7641999999999999e-05 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636786+0000", 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0732000000000001e-05 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636857+0000", 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1418e-05 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.636874+0000", 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6592e-05 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.640439+0000", 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035655249999999999 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.640499+0000", 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9524000000000001e-05 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=125, reqid=client.4526.0:295, at_version=45'45, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:54.197 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.645673+0000", 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.4977650699999998, 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061112989999999997, 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645673+0000", 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645673+0000", 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645674+0000", 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0440000000000001e-06 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645765+0000", 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1082999999999995e-05 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645824+0000", 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9073999999999997e-05 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645838+0000", 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3725e-05 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647211+0000", 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001373193 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647221+0000", 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0575e-05 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651761+0000", 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004540021 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651784+0000", 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2583999999999999e-05 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.201 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.202 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:48:54.202 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.636542+0000", 2024-09-15T02:48:54.203 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.506895616, 2024-09-15T02:48:54.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052986020000000003, 2024-09-15T02:48:54.203 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.203 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.203 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636542+0000", 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636542+0000", 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636544+0000", 2024-09-15T02:48:54.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636982+0000", 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043829999999999997 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637045+0000", 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3392999999999994e-05 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637057+0000", 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1559000000000001e-05 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637114+0000", 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637126+0000", 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1888e-05 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641808+0000", 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046821679999999996 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641841+0000", 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2787000000000003e-05 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 49/33 ECSubWrite(tid=45, reqid=client.4526.0:362, at_version=49'25, trim_to=0'0, roll_forward_to=48'23))", 2024-09-15T02:48:54.207 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.684886+0000", 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.458551345, 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024415959999999999, 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684886+0000", 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684886+0000", 2024-09-15T02:48:54.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684887+0000", 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7000000000000004e-07 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.685148+0000", 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026052099999999999 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.685274+0000", 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012643100000000001 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.685293+0000", 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9085000000000001e-05 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.685410+0000", 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116984 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.685427+0000", 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6320999999999998e-05 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.687247+0000", 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018201389999999999 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.687328+0000", 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1345000000000001e-05 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.211 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.39370836999999997, 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085723550000000003, 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749730+0000", 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1500000000000004e-07 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749937+0000", 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020695800000000001 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750030+0000", 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3487999999999997e-05 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750043+0000", 2024-09-15T02:48:54.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2322e-05 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750162+0000", 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119388 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750182+0000", 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.02e-05 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758264+0000", 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080818000000000001 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758302+0000", 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7583999999999998e-05 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=180, reqid=client.4526.0:385, at_version=49'53, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:48:54.216 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.38383677399999999, 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010517344, 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:48:54.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759603+0000", 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7179999999999999e-06 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759826+0000", 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0002231 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759971+0000", 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014553100000000001 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760158+0000", 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018628199999999999 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760305+0000", 2024-09-15T02:48:54.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014718500000000001 2024-09-15T02:48:54.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760353+0000", 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8418999999999998e-05 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770084+0000", 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097306449999999996 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770118+0000", 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4464000000000001e-05 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.221 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.38255742999999998, 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078090939999999999, 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760882+0000", 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.145e-06 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761368+0000", 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048694100000000001 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761447+0000", 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8586999999999998e-05 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761462+0000", 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4588e-05 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761530+0000", 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8718000000000002e-05 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761543+0000", 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.274e-05 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768640+0000", 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070968000000000003 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768689+0000", 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9574999999999999e-05 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=184, reqid=client.4526.0:386, at_version=49'54, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.767301+0000", 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.37613644800000001, 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033346830000000002, 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.226 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.767301+0000", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.767301+0000", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.767302+0000", 2024-09-15T02:48:54.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.066e-06 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.767358+0000", 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5473999999999998e-05 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.767411+0000", 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3115999999999998e-05 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.767418+0000", 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7020000000000004e-06 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768313+0000", 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089537000000000004 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768328+0000", 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4574e-05 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770608+0000", 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022804879999999998 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770636+0000", 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7892999999999999e-05 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.230 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 49/33 ECSubWrite(tid=48, reqid=client.4526.0:404, at_version=49'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.770958+0000", 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.37247951200000001, 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057426880000000001, 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770958+0000", 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770958+0000", 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770959+0000", 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3600000000000002e-07 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771401+0000", 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044212100000000002 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771443+0000", 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1940999999999997e-05 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771465+0000", 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-15T02:48:54.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771520+0000", 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4901000000000002e-05 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771535+0000", 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4552000000000001e-05 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776642+0000", 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005107388 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776701+0000", 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8829999999999997e-05 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=188, reqid=client.4526.0:389, at_version=49'57, trim_to=0'0, roll_forward_to=49'55))", 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.955686+0000", 2024-09-15T02:48:54.235 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.18775138999999999, 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061702720000000001, 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955686+0000", 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955686+0000", 2024-09-15T02:48:54.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955689+0000", 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3039999999999999e-06 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955797+0000", 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010848099999999999 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955911+0000", 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011391 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955933+0000", 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2039999999999999e-05 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.956065+0000", 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131927 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.956095+0000", 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9589000000000001e-05 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.961823+0000", 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057285050000000001 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.961857+0000", 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3516000000000002e-05 2024-09-15T02:48:54.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.240 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.240 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.240 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.240 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.240 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.241 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:54.235+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:146: done (9 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:148: done (8 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:149: done (7 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:151: done (6 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:152: done (5 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:153: done (4 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:155: done (3 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:156: done (2 left) 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 1 current snap is 9 2024-09-15T02:48:54.242 INFO:tasks.rados.rados.0.smithi191.stdout:157: seq_num 82 ranges {2424832=622592,3047424=475136,3522560=294912} 2024-09-15T02:48:54.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:48:54.251 INFO:tasks.rados.rados.0.smithi191.stdout:157: writing smithi19125844-1 from 2424832 to 3047424 tid 1 2024-09-15T02:48:54.253 INFO:tasks.rados.rados.0.smithi191.stdout:157: writing smithi19125844-1 from 3047424 to 3522560 tid 2 2024-09-15T02:48:54.256 INFO:tasks.rados.rados.0.smithi191.stdout:157: writing smithi19125844-1 from 3522560 to 3817472 tid 3 2024-09-15T02:48:54.256 INFO:tasks.rados.rados.0.smithi191.stdout:158: copy_from oid 27 from oid 28 current snap is 9 2024-09-15T02:48:54.256 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 11 current snap is 9 2024-09-15T02:48:54.256 INFO:tasks.rados.rados.0.smithi191.stdout:159: seq_num 83 ranges {180224=638976,819200=622592,1441792=147456} 2024-09-15T02:48:54.261 INFO:tasks.rados.rados.0.smithi191.stdout:159: writing smithi19125844-11 from 180224 to 819200 tid 1 2024-09-15T02:48:54.266 INFO:tasks.rados.rados.0.smithi191.stdout:159: writing smithi19125844-11 from 819200 to 1441792 tid 2 2024-09-15T02:48:54.267 INFO:tasks.rados.rados.0.smithi191.stdout:159: writing smithi19125844-11 from 1441792 to 1589248 tid 3 2024-09-15T02:48:54.267 INFO:tasks.rados.rados.0.smithi191.stdout:160: rollback oid 50 current snap is 9 2024-09-15T02:48:54.267 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 50 to 4 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:150: finishing copy_from to smithi19125844-8 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 8 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:150: done (5 left) 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:161: read oid 8 snap -1 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:161: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:162: read oid 19 snap -1 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:162: expect (ObjNum 73 snap 7 seq_num 73) 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 41 current snap is 9 2024-09-15T02:48:54.268 INFO:tasks.rados.rados.0.smithi191.stdout:163: seq_num 84 ranges {720896=655360,1376256=638976,2015232=131072} 2024-09-15T02:48:54.276 INFO:tasks.rados.rados.0.smithi191.stdout:163: writing smithi19125844-41 from 720896 to 1376256 tid 1 2024-09-15T02:48:54.281 INFO:tasks.rados.rados.0.smithi191.stdout:163: writing smithi19125844-41 from 1376256 to 2015232 tid 2 2024-09-15T02:48:54.282 INFO:tasks.rados.rados.0.smithi191.stdout:163: writing smithi19125844-41 from 2015232 to 2146304 tid 3 2024-09-15T02:48:54.282 INFO:tasks.rados.rados.0.smithi191.stdout:164: read oid 33 snap -1 2024-09-15T02:48:54.282 INFO:tasks.rados.rados.0.smithi191.stdout:164: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-15T02:48:54.283 INFO:tasks.rados.rados.0.smithi191.stdout:165: delete oid 46 current snap is 9 2024-09-15T02:48:54.283 INFO:tasks.rados.rados.0.smithi191.stdout:158: finishing copy_from racing read to smithi19125844-27 2024-09-15T02:48:54.284 INFO:tasks.rados.rados.0.smithi191.stdout:160: finishing rollback tid 0 to smithi19125844-50 2024-09-15T02:48:54.288 INFO:tasks.rados.rados.0.smithi191.stdout:154: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-15T02:48:54.308 INFO:tasks.rados.rados.0.smithi191.stdout:159: finishing write tid 1 to smithi19125844-11 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:158: finishing copy_from to smithi19125844-27 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 27 v 4 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:157: finishing write tid 1 to smithi19125844-1 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:157: finishing write tid 2 to smithi19125844-1 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:157: finishing write tid 3 to smithi19125844-1 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:160: finishing rollback tid 1 to smithi19125844-50 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:160: finishing rollback tid 2 to smithi19125844-50 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-15T02:48:54.309 INFO:tasks.rados.rados.0.smithi191.stdout:161: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T02:48:54.314 INFO:tasks.rados.rados.0.smithi191.stdout:162: expect (ObjNum 73 snap 7 seq_num 73) 2024-09-15T02:48:54.342 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:54.337+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:54.355 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.355 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:54.355 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:48:54.355 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:48:54.403 INFO:tasks.rados.rados.0.smithi191.stdout:159: finishing write tid 2 to smithi19125844-11 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:159: finishing write tid 3 to smithi19125844-11 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:159: finishing write tid 4 to smithi19125844-11 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:159: finishing write tid 5 to smithi19125844-11 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 11 v 7 (ObjNum 83 snap 9 seq_num 83) dirty exists 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:159: left oid 11 (ObjNum 83 snap 9 seq_num 83) 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:163: finishing write tid 1 to smithi19125844-41 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:163: finishing write tid 2 to smithi19125844-41 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:163: finishing write tid 3 to smithi19125844-41 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:157: finishing write tid 4 to smithi19125844-1 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:157: finishing write tid 5 to smithi19125844-1 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 1 v 35 (ObjNum 82 snap 9 seq_num 82) dirty exists 2024-09-15T02:48:54.404 INFO:tasks.rados.rados.0.smithi191.stdout:157: left oid 1 (ObjNum 82 snap 9 seq_num 82) 2024-09-15T02:48:54.443 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:54.435+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:54.458 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.459 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:54.459 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:48:54.459 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:48:54.459 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:48:54.487 INFO:tasks.rados.rados.0.smithi191.stdout:164: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-15T02:48:54.544 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:54.542+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:54.554 INFO:tasks.rados.rados.0.smithi191.stdout:163: finishing write tid 4 to smithi19125844-41 2024-09-15T02:48:54.556 INFO:tasks.rados.rados.0.smithi191.stdout:154: done (9 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:157: done (8 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:158: done (7 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:159: done (6 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:160: done (5 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:161: done (4 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:162: done (3 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:164: done (2 left) 2024-09-15T02:48:54.557 INFO:tasks.rados.rados.0.smithi191.stdout:165: done (1 left) 2024-09-15T02:48:54.558 INFO:tasks.rados.rados.0.smithi191.stdout:166: snap_create 2024-09-15T02:48:54.559 INFO:tasks.rados.rados.0.smithi191.stdout:163: finishing write tid 5 to smithi19125844-41 2024-09-15T02:48:54.559 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 41 v 62 (ObjNum 84 snap 9 seq_num 84) dirty exists 2024-09-15T02:48:54.559 INFO:tasks.rados.rados.0.smithi191.stdout:163: left oid 41 (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:48:54.575 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:48:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:48:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:48:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4311.0:5 2.2 e28/26 2:5cea7035:::rbd_info:head v 28'1, mlcod=0'0)", 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:28.407879+0000", 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.165280366000001, 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001584163, 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.407879+0000", 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.407879+0000", 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.407880+0000", 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9699999999999996e-07 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.407889+0000", 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8599999999999999e-06 2024-09-15T02:48:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.407974+0000", 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5434999999999998e-05 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.407994+0000", 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9398999999999998e-05 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.408136+0000", 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014263399999999999 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.408176+0000", 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9375999999999997e-05 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.409410+0000", 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012347930000000001 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:28.409463+0000", 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3069000000000003e-05 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4311.0:7 2.2 e29/26 2:5cea7035:::rbd_info:head v 29'2, mlcod=0'0)", 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.150741699000001, 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017517679999999999, 2024-09-15T02:48:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422427+0000", 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.473e-06 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422543+0000", 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115732 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422561+0000", 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8505e-05 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422669+0000", 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108175 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422706+0000", 2024-09-15T02:48:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6851999999999999e-05 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.424081+0000", 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013743970000000001 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.424170+0000", 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9098000000000007e-05 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 39/33 ECSubWrite(tid=16, reqid=client.4526.0:231, at_version=39'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.999376746999999, 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00223799, 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:48:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9699999999999996e-07 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573829+0000", 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5249e-05 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573916+0000", 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6917999999999997e-05 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573931+0000", 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5546e-05 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574047+0000", 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011535300000000001 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574060+0000", 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3016999999999999e-05 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575974+0000", 2024-09-15T02:48:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019143750000000001 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576021+0000", 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6934999999999997e-05 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 39/33 ECSubWrite(tid=17, reqid=client.4526.0:232, at_version=39'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:48:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.574020+0000", 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.999139395, 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025056259999999999, 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574020+0000", 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574020+0000", 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574021+0000", 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1200000000000002e-07 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574044+0000", 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3465999999999999e-05 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574087+0000", 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2979999999999998e-05 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574096+0000", 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5969999999999992e-06 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575374+0000", 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00127786 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575390+0000", 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5619000000000001e-05 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576491+0000", 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011011459999999999 2024-09-15T02:48:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576526+0000", 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5145999999999999e-05 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 39/33 ECSubWrite(tid=19, reqid=client.4526.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.057423+0000", 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.515736693000001, 2024-09-15T02:48:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001707879, 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057423+0000", 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057423+0000", 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057425+0000", 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9570000000000001e-06 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057452+0000", 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6959000000000001e-05 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057544+0000", 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1885000000000005e-05 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057562+0000", 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8669e-05 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057693+0000", 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013111399999999999 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.057721+0000", 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7246e-05 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.059065+0000", 2024-09-15T02:48:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001344051 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.059131+0000", 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5998000000000001e-05 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=20, reqid=client.4526.0:249, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.9807529079999995, 2024-09-15T02:48:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020019123999999999, 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:48:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592951+0000", 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054339899999999997 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593028+0000", 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6892999999999997e-05 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593044+0000", 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6407999999999999e-05 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595661+0000", 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002617032 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595686+0000", 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5358999999999999e-05 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612387+0000", 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016700194000000002 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612426+0000", 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9138999999999999e-05 2024-09-15T02:48:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=21, reqid=client.4526.0:250, at_version=41'14, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.9688401570000007, 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086779219999999994, 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604321+0000", 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.902e-06 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604517+0000", 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019603099999999999 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604572+0000", 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4144000000000002e-05 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604581+0000", 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4360000000000008e-06 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608062+0000", 2024-09-15T02:48:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034810710000000001 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608079+0000", 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7059999999999999e-05 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612964+0000", 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004885197 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612997+0000", 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3080999999999997e-05 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=22, reqid=client.4526.0:251, at_version=41'15, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.608374+0000", 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.9647857579999997, 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049527460000000001, 2024-09-15T02:48:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608374+0000", 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608374+0000", 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608375+0000", 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608397+0000", 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1888000000000001e-05 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608443+0000", 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6020000000000003e-05 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608451+0000", 2024-09-15T02:48:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.456e-06 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612489+0000", 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004037629 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612499+0000", 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0363000000000001e-05 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.613275+0000", 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077597099999999997 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.613327+0000", 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1484999999999999e-05 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/33 ECSubWrite(tid=36, reqid=client.4526.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=43'21))", 2024-09-15T02:48:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.372910+0000", 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.2002491500000003, 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018596559999999999, 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372910+0000", 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372910+0000", 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372912+0000", 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8470000000000001e-06 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372948+0000", 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5621999999999999e-05 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373050+0000", 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101875 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373069+0000", 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8706e-05 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373225+0000", 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015610600000000001 2024-09-15T02:48:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373260+0000", 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4965999999999997e-05 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.374697+0000", 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001437345 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.374770+0000", 2024-09-15T02:48:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3188999999999993e-05 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=44'21, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9750198690000005, 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004464193, 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-15T02:48:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598197+0000", 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6575e-05 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598305+0000", 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010784400000000001 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598319+0000", 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3987e-05 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598448+0000", 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000129481 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598479+0000", 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0629000000000001e-05 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602507+0000", 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040278609999999998 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602604+0000", 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7145999999999995e-05 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9731934820000001, 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080113409999999999, 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:48:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599967+0000", 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3120000000000001e-06 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600013+0000", 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126999999999997e-05 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600081+0000", 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8295999999999998e-05 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600094+0000", 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3522e-05 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600358+0000", 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026319700000000001 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600388+0000", 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029e-05 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607948+0000", 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075605949999999998 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607977+0000", 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9263000000000001e-05 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=31, reqid=osd.8.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:48:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.609028+0000", 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9641312539999998, 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023506960000000002, 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609028+0000", 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609028+0000", 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609030+0000", 2024-09-15T02:48:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1969999999999999e-06 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609043+0000", 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3813e-05 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609098+0000", 2024-09-15T02:48:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5025999999999998e-05 2024-09-15T02:48:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609108+0000", 2024-09-15T02:48:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2809999999999997e-06 2024-09-15T02:48:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609631+0000", 2024-09-15T02:48:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052322800000000004 2024-09-15T02:48:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609641+0000", 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0206e-05 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.611349+0000", 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017074239999999999 2024-09-15T02:48:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.611379+0000", 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0521000000000001e-05 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/33 ECSubWrite(tid=39, reqid=osd.14.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=44'25))", 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.346169+0000", 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.226990958, 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022908160000000002, 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346169+0000", 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346169+0000", 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346171+0000", 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069999999999998e-06 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346195+0000", 2024-09-15T02:48:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4604999999999999e-05 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346288+0000", 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2881999999999996e-05 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346304+0000", 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5373e-05 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346473+0000", 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000169603 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346507+0000", 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3442000000000003e-05 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.348384+0000", 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018776820000000001 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.348459+0000", 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5022000000000001e-05 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.9364857739999999, 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050988680000000003, 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:48:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2399999999999999e-07 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637113+0000", 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043827200000000002 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637157+0000", 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4824999999999999e-05 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637167+0000", 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7950000000000002e-06 2024-09-15T02:48:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637247+0000", 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0315000000000006e-05 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637263+0000", 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5156e-05 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641738+0000", 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044751230000000001 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641773+0000", 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4958e-05 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 50/33 ECSubWrite(tid=36, reqid=client.4526.0:421, at_version=50'7, trim_to=0'0, roll_forward_to=36'5))", 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.269071+0000", 2024-09-15T02:48:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.30408845000000001, 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022646060000000002, 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269071+0000", 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269071+0000", 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269072+0000", 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4799999999999999e-07 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269131+0000", 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9839000000000001e-05 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:48:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3114999999999994e-05 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269217+0000", 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3029e-05 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269306+0000", 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8188000000000003e-05 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269325+0000", 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9264000000000001e-05 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271283+0000", 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019582879999999999 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271336+0000", 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2534999999999998e-05 2024-09-15T02:48:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.30226884599999998, 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012340179999999999, 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:48:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0999999999999999e-07 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271155+0000", 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026426800000000001 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271179+0000", 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3893999999999999e-05 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271190+0000", 2024-09-15T02:48:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0525e-05 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271256+0000", 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6328999999999995e-05 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271268+0000", 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1831000000000001e-05 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283197+0000", 2024-09-15T02:48:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011929294999999999 2024-09-15T02:48:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283231+0000", 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3627999999999997e-05 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.278363+0000", 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.29479702800000002, 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051390910000000001, 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278363+0000", 2024-09-15T02:48:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278363+0000", 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278364+0000", 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.139e-06 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278542+0000", 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017818700000000001 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278581+0000", 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9258999999999998e-05 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278588+0000", 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3220000000000004e-06 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278780+0000", 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191681 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278787+0000", 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3990000000000001e-06 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283463+0000", 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046766899999999998 2024-09-15T02:48:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283502+0000", 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8414000000000002e-05 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=62, reqid=client.4526.0:418, at_version=50'6, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284004+0000", 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.28915515200000003, 2024-09-15T02:48:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032969980000000002, 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284004+0000", 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284004+0000", 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284006+0000", 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284120+0000", 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114286 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284187+0000", 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7119000000000006e-05 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284195+0000", 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0639999999999994e-06 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284257+0000", 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1577999999999999e-05 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284270+0000", 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3038e-05 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287270+0000", 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003000674 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287301+0000", 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0994000000000002e-05 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 50/33 ECSubWrite(tid=37, reqid=client.4526.0:422, at_version=50'8, trim_to=0'0, roll_forward_to=36'5))", 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284545+0000", 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.288614494, 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001520694, 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284545+0000", 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284545+0000", 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284546+0000", 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8199999999999998e-07 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284572+0000", 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6299999999999999e-05 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284647+0000", 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4814999999999994e-05 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284655+0000", 2024-09-15T02:48:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5780000000000001e-06 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284713+0000", 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8517000000000002e-05 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284725+0000", 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1389000000000001e-05 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286038+0000", 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001313112 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286066+0000", 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8101000000000001e-05 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 50/33 ECSubWrite(tid=38, reqid=client.4526.0:423, at_version=50'9, trim_to=0'0, roll_forward_to=36'5))", 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284577+0000", 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.28858255900000002, 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015035269999999999, 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284577+0000", 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284577+0000", 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284578+0000", 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999999e-07 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:54.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284602+0000", 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4096000000000001e-05 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284680+0000", 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7945e-05 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284685+0000", 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2469999999999999e-06 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.285243+0000", 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055788400000000003 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.285252+0000", 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7969999999999995e-06 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:54.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286069+0000", 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081654700000000004 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286081+0000", 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1539000000000001e-05 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:48:54.678 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:54.674+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:54.778 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:54.773+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:54.785 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.785 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:54.785 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:48:54.785 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:48:54.879 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:54.877+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:54.897 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:54.897 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:54.897 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:48:54.897 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:48:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:54.907 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:48:54.980 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:54.977+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:222 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.566500+0000", 2024-09-15T02:48:55.006 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.436301330999999, 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035857158, 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 222 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566500+0000", 2024-09-15T02:48:55.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566500+0000", 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566502+0000", 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9479999999999998e-06 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.566986+0000", 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048457099999999999 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567013+0000", 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6429999999999999e-05 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567027+0000", 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4246e-05 2024-09-15T02:48:55.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578885+0000", 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011858472 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.578982+0000", 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6564000000000006e-05 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.579827+0000", 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084486900000000005 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583771+0000", 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039440459999999997 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602292+0000", 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018520688 2024-09-15T02:48:55.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602357+0000", 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5324e-05 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:223 3.1s0 3:8610d5f9:::smithi19125844-1:head [setxattr _header (55) in=7b] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567075+0000", 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.435726301000001, 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035541204, 2024-09-15T02:48:55.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 223 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567075+0000", 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567075+0000", 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567076+0000", 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.437e-06 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567083+0000", 2024-09-15T02:48:55.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8730000000000001e-06 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567098+0000", 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.484e-05 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567107+0000", 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3540000000000006e-06 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582906+0000", 2024-09-15T02:48:55.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015798525000000001 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.582986+0000", 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0675999999999998e-05 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583388+0000", 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040186600000000002 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583848+0000", 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046005200000000002 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602517+0000", 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018668657000000002 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.602616+0000", 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8924000000000004e-05 2024-09-15T02:48:55.017 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:224 3.1s0 3:8610d5f9:::smithi19125844-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.567193+0000", 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.435608452, 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037648236000000002, 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 224 2024-09-15T02:48:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567193+0000", 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567193+0000", 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567194+0000", 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.595e-06 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567199+0000", 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567212+0000", 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2969000000000001e-05 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.567221+0000", 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4550000000000001e-06 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583610+0000", 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389053000000001 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.583714+0000", 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103657 2024-09-15T02:48:55.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.603148+0000", 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019434276 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.603245+0000", 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7081000000000001e-05 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.604841+0000", 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001595679 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:388 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.2500008789999999, 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.205927366, 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.023 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 388 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752805+0000", 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8480000000000003e-06 2024-09-15T02:48:55.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752820+0000", 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752827+0000", 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8059999999999997e-06 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770449+0000", 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017621527000000001 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770567+0000", 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118576 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776872+0000", 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063044340000000003 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776986+0000", 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011429699999999999 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958728+0000", 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.181741654 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:389 3.7s0 3:e10d9d4d:::smithi19125844-8:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.2499656180000001, 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47945425899999999, 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 389 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.029 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752836+0000", 2024-09-15T02:48:55.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752842+0000", 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8579999999999996e-06 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752860+0000", 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7816e-05 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752867+0000", 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0260000000000002e-06 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770793+0000", 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017925963999999999 2024-09-15T02:48:55.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770931+0000", 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137978 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955584+0000", 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.184652765 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.959571+0000", 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003987488 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232070+0000", 2024-09-15T02:48:55.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.272499243 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232290+0000", 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021937500000000001 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:390 3.7s0 3:e10d9d4d:::smithi19125844-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:48:55.035 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.2499274279999999, 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20558775300000001, 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 390 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8199999999999999e-07 2024-09-15T02:48:55.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752878+0000", 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4489999999999999e-06 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752888+0000", 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0122000000000001e-05 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752894+0000", 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9220000000000002e-06 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771143+0000", 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018248665000000001 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771198+0000", 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5158000000000003e-05 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958258+0000", 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18706018699999999 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958359+0000", 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101025 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958461+0000", 2024-09-15T02:48:55.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102543 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:402 3.1s0 3:801a1da7:::smithi19125844-39:e [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.2445496650000001, 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52960550699999998, 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.041 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 402 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758252+0000", 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5000000000000002e-07 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758257+0000", 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3800000000000004e-06 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758267+0000", 2024-09-15T02:48:55.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0455000000000001e-05 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758604+0000", 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033705300000000003 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761065+0000", 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024613840000000001 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761208+0000", 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014282499999999999 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287857+0000", 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52664865999999999 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.045 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 49/33 ECSubReadReply(tid=183, attrs_read=0))", 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.239577165, 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52471420199999996, 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:48:55.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:48:55.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763572+0000", 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034791800000000001 2024-09-15T02:48:55.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286378+0000", 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52280558799999999 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286384+0000", 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286558+0000", 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017370499999999999 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287938+0000", 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013805550000000001 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:409 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.251674+0000", 2024-09-15T02:48:55.050 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.75112710999999999, 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040551622000000002, 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 409 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.251674+0000", 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.251674+0000", 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.251675+0000", 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5030000000000001e-06 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252402+0000", 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00072673699999999998 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252427+0000", 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4392999999999998e-05 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252439+0000", 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2072000000000001e-05 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252564+0000", 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012556500000000001 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252798+0000", 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023401699999999999 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254690+0000", 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018922279999999999 2024-09-15T02:48:55.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269207+0000", 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014516625999999999 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292131+0000", 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022924243 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292226+0000", 2024-09-15T02:48:55.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4238000000000001e-05 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:410 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~475136] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.254301+0000", 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.74850005200000003, 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038330291000000002, 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.056 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 410 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254301+0000", 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254301+0000", 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254302+0000", 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1999999999999999e-06 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254921+0000", 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061855899999999997 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254944+0000", 2024-09-15T02:48:55.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3079999999999999e-05 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254954+0000", 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9229999999999997e-06 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.259708+0000", 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047539380000000001 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.259955+0000", 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000246977 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.261701+0000", 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017462129999999999 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.274849+0000", 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013147684 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292368+0000", 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017519525000000001 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292631+0000", 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026319199999999998 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:411 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~294912] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.256170+0000", 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.74663086700000003, 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036676868000000001, 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 411 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256170+0000", 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256170+0000", 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256172+0000", 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.499e-06 2024-09-15T02:48:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256516+0000", 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034406600000000002 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256537+0000", 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1032000000000001e-05 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256547+0000", 2024-09-15T02:48:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0468e-05 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269243+0000", 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012695778 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269331+0000", 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7598999999999999e-05 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270076+0000", 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074561500000000004 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277963+0000", 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078864150000000008 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292776+0000", 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014813692 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292847+0000", 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0704000000000006e-05 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:412 3.1s0 3:8610d5f9:::smithi19125844-1:head [setxattr _header (55) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.256557+0000", 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.74624439300000001, 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.061249245000000001, 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.067 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 412 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256557+0000", 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256557+0000", 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256558+0000", 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7400000000000002e-07 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256563+0000", 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1170000000000003e-06 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256572+0000", 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7210000000000006e-06 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256579+0000", 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3149999999999997e-06 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276049+0000", 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019470022999999999 2024-09-15T02:48:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276235+0000", 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018618899999999999 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276938+0000", 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000703333 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277968+0000", 2024-09-15T02:48:55.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001029807 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.317677+0000", 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039708808999999998 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.317806+0000", 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012905699999999999 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:413 3.1s0 3:8610d5f9:::smithi19125844-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.74621552800000002, 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063143321000000002, 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.073 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 413 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256595+0000", 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.473e-06 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256608+0000", 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3523000000000001e-05 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256615+0000", 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8480000000000003e-06 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278005+0000", 2024-09-15T02:48:55.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021389925000000001 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278191+0000", 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018626800000000001 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.318427+0000", 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040235663999999997 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.318542+0000", 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011523000000000001 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319729+0000", 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011866540000000001 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:433 3.7s0 3:ec23d144:::smithi19125844-41:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.078 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.719807792, 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27191602199999998, 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282994+0000", 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1790000000000001e-06 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283000+0000", 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0479999999999998e-06 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283008+0000", 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1440000000000003e-06 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283012+0000", 2024-09-15T02:48:55.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7129999999999996e-06 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298201+0000", 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015188495 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298355+0000", 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015441200000000001 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298951+0000", 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059618300000000002 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300677+0000", 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001725285 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554804+0000", 2024-09-15T02:48:55.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25412710900000002 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554909+0000", 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105454 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:434 3.7s0 3:ec23d144:::smithi19125844-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:48:55.084 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.71978386900000002, 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27651220700000001, 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 434 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283018+0000", 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0500000000000003e-07 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283021+0000", 2024-09-15T02:48:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.959e-06 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283028+0000", 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0169999999999999e-06 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283032+0000", 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0509999999999998e-06 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299373+0000", 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016340659 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299481+0000", 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108218 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554991+0000", 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25551037700000001 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.555256+0000", 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026495400000000002 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559529+0000", 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042733670000000001 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:437 3.6s0 3:6e9f7d10:::smithi19125844-33:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.71971693199999998, 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20344801100000001, 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 437 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283085+0000", 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283088+0000", 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.993e-06 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283094+0000", 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5810000000000002e-06 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6289999999999999e-06 2024-09-15T02:48:55.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284035+0000", 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093763400000000002 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284100+0000", 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4882999999999995e-05 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486532+0000", 2024-09-15T02:48:55.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20243185599999999 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:438 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [delete] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e50)", 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.71968312300000004, 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27354936000000002, 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:48:55.094 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 438 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8100000000000003e-07 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283121+0000", 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8830000000000002e-06 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283129+0000", 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3390000000000004e-06 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283133+0000", 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299523+0000", 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389989000000001 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299625+0000", 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010208 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300116+0000", 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049149000000000003 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300972+0000", 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085554200000000004 2024-09-15T02:48:55.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556582+0000", 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25561025399999998 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556667+0000", 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5208000000000001e-05 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.6s0 50/33 ECSubReadReply(tid=200, attrs_read=0))", 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.70751352999999995, 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19127902199999999, 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2600000000000001e-07 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295338+0000", 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9951999999999999e-05 2024-09-15T02:48:55.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485520+0000", 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19018270600000001 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485530+0000", 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0820000000000005e-06 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485655+0000", 2024-09-15T02:48:55.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012580999999999999 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486567+0000", 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000911246 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=203, last_complete=50'62, committed=1, applied=1))", 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:48:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.69946834800000002, 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25159346799999999, 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:48:55.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303336+0000", 2024-09-15T02:48:55.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9550000000000001e-06 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554637+0000", 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25130088299999997 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554647+0000", 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9329999999999996e-06 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554704+0000", 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6514000000000001e-05 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554926+0000", 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222647 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 50/33 ECSubWriteReply(tid=205, last_complete=50'35, committed=1, applied=1))", 2024-09-15T02:48:55.107 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.68346091799999997, 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.23186345, 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:48:55.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319341+0000", 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6700000000000003e-07 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319347+0000", 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6879999999999996e-06 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550925+0000", 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.231578591 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550946+0000", 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0903e-05 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551110+0000", 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016408300000000001 2024-09-15T02:48:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551204+0000", 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3418000000000006e-05 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:48:55.111 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:55.113 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:48:55.113 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:48:55.109+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:48:55.215 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:55.209+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:55.225 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:55.225 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:55.226 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:55.226 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:55.231 INFO:tasks.rados.rados.0.smithi191.stdout:163: done (1 left) 2024-09-15T02:48:55.231 INFO:tasks.rados.rados.0.smithi191.stdout:166: done (0 left) 2024-09-15T02:48:55.231 INFO:tasks.rados.rados.0.smithi191.stdout:167: rollback oid 7 current snap is 10 2024-09-15T02:48:55.232 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 7 to 8 2024-09-15T02:48:55.232 INFO:tasks.rados.rados.0.smithi191.stdout:168: rollback oid 11 current snap is 10 2024-09-15T02:48:55.232 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 11 to 9 2024-09-15T02:48:55.232 INFO:tasks.rados.rados.0.smithi191.stdout:169: read oid 4 snap -1 2024-09-15T02:48:55.232 INFO:tasks.rados.rados.0.smithi191.stdout:169: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-15T02:48:55.233 INFO:tasks.rados.rados.0.smithi191.stdout:170: snap_create 2024-09-15T02:48:55.235 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:48:55.240 INFO:tasks.rados.rados.0.smithi191.stdout:167: finishing rollback tid 0 to smithi19125844-7 2024-09-15T02:48:55.241 INFO:tasks.rados.rados.0.smithi191.stdout:167: finishing rollback tid 1 to smithi19125844-7 2024-09-15T02:48:55.242 INFO:tasks.rados.rados.0.smithi191.stdout:167: finishing rollback tid 2 to smithi19125844-7 2024-09-15T02:48:55.242 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 7 v 18 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-15T02:48:55.243 INFO:tasks.rados.rados.0.smithi191.stdout:169: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-15T02:48:55.245 INFO:tasks.rados.rados.0.smithi191.stdout:168: finishing rollback tid 0 to smithi19125844-11 2024-09-15T02:48:55.245 INFO:tasks.rados.rados.0.smithi191.stdout:168: finishing rollback tid 1 to smithi19125844-11 2024-09-15T02:48:55.245 INFO:tasks.rados.rados.0.smithi191.stdout:168: finishing rollback tid 2 to smithi19125844-11 2024-09-15T02:48:55.245 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 11 v 11 (ObjNum 83 snap 9 seq_num 83) dirty exists 2024-09-15T02:48:55.315 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:55.307+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:55.323 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:55.323 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:55.323 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:55.323 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:55.324 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:55.332 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:48:55.416 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:55.413+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 30.697069933000002, 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.438 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:48:55.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:48:55.442 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:220 3.es0 3:71c1a91c:::smithi19125844-45:head [create,setxattr YOk\\Qsqnjk`Uh[icZasWhcZnad[h[k[Qmk]TihbR (1823) in=40b,setxattr QnfshqY (34) in=7b,setxattr V_tZOrZWhpPekPNs_toQaraRlW\\sWffpapfPROXb (1614) in=40b,setxattr aiTbV]hXcpXjkS]VQonZZkfogbUSnO[bajbZWdRf (839) in=40b,setxattr pZdlecZrV]qp]`sTkf_bSjZVZjhbScXfb[Sr_bsa (18524) in=40b,setxattr kXtOdjofgYOa_Tck\\nYsbVOfgtldphnQ`]P[j]lX (1295) in=40b,setxattr dRpscRhc^OtlbThWSmXVSNge]s (34) in=26b,setxattr T\\s[QdNhlRZh\\\\[X]NpSccmfRcqRaXScZOZ\\ZR^e (1999) in=40b,setxattr YTcfnqlbrRa^rhctrncU_Wa]laamPmYsaqg_f^N` (1471) in=40b,setxattr l_Qeh\\RcafrRYN_YZf_iiaWagZfe[sop]^\\qTggq (561) in=40b,setxattr NShVOQe\\dRYY[tPettNbkgkTVonNshnbYeePWcla (314) in=40b,setxattr ^iVidVQX]Yn`XTY\\pOXm`_PNX`qeckhbQoR`aQWZ (8841) in=40b,setxattr _jkmO]cXrN_S^act\\QOOjpUZOj_Qf^otnZbpWZPU (981) in=40b,setxattr bNbhPX[YP[lppmc`PkWcpnS`XsQNpW\\UYnWr\\_RX (1927) in=40b,setxattr talTebhpYS[[TXcgodoQR\\tf`fVekppjlkWnTqf[ (1753) in=40b,setxattr rp\\XTgd\\X^qaodkhY[elq^RagRhN_NR^j^NmjOUh (806) in=40b,setxattr ]\\oXSo`PWafUO^dbhR^i`hifQgWjRhR[rRa^bnej (70) in=40b,setxattr UkYrdYaZakpnReakOdc\\^bO[YPqfdkdtbNkghaSQ (279) in=40b,setxattr YtUs_hcRdQqgZV]NaadOflcb\\_c^\\Zg_Y\\]snXXZ (701) in=40b,setxattr NRZdh]WclahTe^_btdqe\\^mVYqQOaXaa\\aPVXWnk (19016) in=40b] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.554086+0000", 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.879264311, 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0098341899999999996, 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.443 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 220 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554086+0000", 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554086+0000", 2024-09-15T02:48:55.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554089+0000", 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5670000000000002e-06 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554222+0000", 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013293199999999999 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554249+0000", 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7223e-05 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554262+0000", 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3285000000000001e-05 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554370+0000", 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010724499999999999 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.554397+0000", 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7356999999999999e-05 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.558117+0000", 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00372048 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.558280+0000", 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016244400000000001 2024-09-15T02:48:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.560050+0000", 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017700280000000001 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.561521+0000", 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001470741 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.563801+0000", 2024-09-15T02:48:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002279934 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.563921+0000", 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000119954 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.568541+0000", 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.864809588, 2024-09-15T02:48:55.449 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013583105, 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568541+0000", 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568541+0000", 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.450 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.568542+0000", 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3099999999999996e-07 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570537+0000", 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0019946109999999999 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570648+0000", 2024-09-15T02:48:55.451 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000111309 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570656+0000", 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5569999999999996e-06 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570713+0000", 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6875e-05 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570722+0000", 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8349999999999993e-06 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582100+0000", 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011377395 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582124+0000", 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4692e-05 2024-09-15T02:48:55.453 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:301 3.es0 3:71c1a91c:::smithi19125844-45:head [delete] snapc a=[a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45)", 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:49.632452+0000", 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.8008986260000004, 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083448659999999994, 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.454 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 301 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632452+0000", 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632452+0000", 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632455+0000", 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0900000000000001e-06 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632468+0000", 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2767e-05 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632508+0000", 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9706000000000002e-05 2024-09-15T02:48:55.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632528+0000", 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9910000000000001e-05 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632646+0000", 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000118267 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.632830+0000", 2024-09-15T02:48:55.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018412100000000001 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.635359+0000", 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0025289710000000001 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.639460+0000", 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0041008759999999998 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.640723+0000", 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012635050000000001 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.640797+0000", 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3652999999999994e-05 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:48:55.459 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:50.649757+0000", 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.783593261, 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016275881999999998, 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.649757+0000", 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.460 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.649757+0000", 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.649758+0000", 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650149+0000", 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00039158200000000002 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650186+0000", 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6442000000000001e-05 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650194+0000", 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0879999999999995e-06 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650310+0000", 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000116174 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650325+0000", 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4946000000000001e-05 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.665984+0000", 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015658647000000001 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.666033+0000", 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9663000000000002e-05 2024-09-15T02:48:55.463 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/33 ECSubWrite(tid=164, reqid=client.4526.0:313, at_version=46'24, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:50.657528+0000", 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.7758223329999998, 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0091179179999999992, 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.464 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.657528+0000", 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.657528+0000", 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.657530+0000", 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.449e-06 2024-09-15T02:48:55.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.657739+0000", 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000209683 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.657798+0000", 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8341000000000003e-05 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.657808+0000", 2024-09-15T02:48:55.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0611999999999999e-05 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.662124+0000", 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043151810000000004 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.662134+0000", 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0362e-05 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.666610+0000", 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0044757340000000003 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.666646+0000", 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6556e-05 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:48:55.468 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.691715+0000", 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.7416355530000001, 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010201847, 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.691715+0000", 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.691715+0000", 2024-09-15T02:48:55.469 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.691716+0000", 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2600000000000002e-07 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692131+0000", 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00041543400000000001 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692168+0000", 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7067000000000003e-05 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692174+0000", 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9059999999999999e-06 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692228+0000", 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3825000000000001e-05 2024-09-15T02:48:55.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692241+0000", 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2481e-05 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.701881+0000", 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096407100000000003 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.701917+0000", 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5698000000000003e-05 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:403 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~606208] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.473 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.676331239, 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.195887642, 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 403 2024-09-15T02:48:55.474 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757022+0000", 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.593e-06 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757800+0000", 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077824999999999997 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757830+0000", 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0057e-05 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757844+0000", 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3947999999999999e-05 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757925+0000", 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0249999999999999e-05 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.758082+0000", 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015777599999999999 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759386+0000", 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001303217 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777348+0000", 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017962568000000002 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952681+0000", 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17533264000000001 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952907+0000", 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000226343 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.479 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:404 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~720896] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.672112635, 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.192160255, 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 404 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.480 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761280+0000", 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2138999999999998e-05 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762400+0000", 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001119451 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762433+0000", 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3238000000000003e-05 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762446+0000", 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3308e-05 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768118+0000", 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056721360000000004 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768285+0000", 2024-09-15T02:48:55.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000166482 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769662+0000", 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013768140000000001 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780475+0000", 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010813578000000001 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953275+0000", 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17280025199999999 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953398+0000", 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012285700000000001 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:405 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~49152] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.6708818970000001, 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19129260000000001, 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 405 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.486 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762470+0000", 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4699999999999999e-06 2024-09-15T02:48:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762527+0000", 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7151999999999998e-05 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762550+0000", 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2311999999999999e-05 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762559+0000", 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777614+0000", 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015055628999999999 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777696+0000", 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1840000000000002e-05 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778120+0000", 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042372900000000002 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780527+0000", 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00240712 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953622+0000", 2024-09-15T02:48:55.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17309514500000001 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953761+0000", 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013926300000000001 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:406 3.es0 3:71c1a91c:::smithi19125844-45:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.6707819370000001, 2024-09-15T02:48:55.491 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19225968700000001, 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 406 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.492 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762570+0000", 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.034e-06 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762575+0000", 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4079999999999997e-06 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762592+0000", 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.715e-05 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762600+0000", 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6480000000000003e-06 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779660+0000", 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017060035000000001 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779757+0000", 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6537000000000001e-05 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780137+0000", 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038070299999999998 2024-09-15T02:48:55.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781414+0000", 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012764230000000001 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954649+0000", 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17323481900000001 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954828+0000", 2024-09-15T02:48:55.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017992999999999999 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:407 3.es0 3:71c1a91c:::smithi19125844-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.670742065, 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19532122900000001, 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:55.497 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 407 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.498 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762610+0000", 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762614+0000", 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0659999999999997e-06 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.499 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762626+0000", 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2490000000000001e-05 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762639+0000", 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2704000000000001e-05 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781806+0000", 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019166823999999999 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.500 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.782082+0000", 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027635000000000002 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955680+0000", 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17359843999999999 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955807+0000", 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012658299999999999 2024-09-15T02:48:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.957930+0000", 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021228499999999999 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.es0 49/33 ECSubWriteReply(tid=47, last_complete=49'26, committed=1, applied=1))", 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.767666+0000", 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.6656849230000002, 2024-09-15T02:48:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095927389999999994, 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.767666+0000", 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.767666+0000", 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.767667+0000", 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5799999999999999e-06 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.767676+0000", 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1120000000000002e-06 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.767751+0000", 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4747999999999997e-05 2024-09-15T02:48:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.767764+0000", 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2894999999999999e-05 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777237+0000", 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0094731610000000008 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777259+0000", 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1243e-05 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.13.22:5 3.es0 3:740f3d5b:::smithi19125844-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.658211085, 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.177222558, 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775141+0000", 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:48:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775146+0000", 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5630000000000002e-06 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775159+0000", 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3509999999999999e-05 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775165+0000", 2024-09-15T02:48:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8130000000000001e-06 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778738+0000", 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003572867 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778948+0000", 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00021028299999999999 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952362+0000", 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17341393199999999 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.253649+0000", 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.179701401, 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016800161000000001, 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.253649+0000", 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.253649+0000", 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.253650+0000", 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7299999999999995e-07 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.254149+0000", 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00049936800000000003 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.254210+0000", 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0278000000000001e-05 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.254225+0000", 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5316999999999999e-05 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.254350+0000", 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012526700000000001 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1984000000000001e-05 2024-09-15T02:48:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270418+0000", 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016055653999999999 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270449+0000", 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1619999999999999e-05 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.514 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.260527+0000", 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.1728240560000001, 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021017170000000002, 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260527+0000", 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.515 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260527+0000", 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260528+0000", 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.55e-06 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260685+0000", 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000156646 2024-09-15T02:48:55.516 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260761+0000", 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5753000000000005e-05 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260774+0000", 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3132999999999999e-05 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.264984+0000", 2024-09-15T02:48:55.517 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042104040000000001 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.265024+0000", 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0046999999999998e-05 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281494+0000", 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016469609 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.518 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281544+0000", 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0028000000000003e-05 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 50/33 ECSubWrite(tid=195, reqid=client.4526.0:411, at_version=50'34, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.269054+0000", 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.1642970960000001, 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013223558999999999, 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.519 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.269054+0000", 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.269054+0000", 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.269055+0000", 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3349999999999999e-06 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.520 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.269127+0000", 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2274999999999995e-05 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.269188+0000", 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0562000000000001e-05 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.269198+0000", 2024-09-15T02:48:55.521 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7540000000000001e-06 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270566+0000", 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013681660000000001 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270586+0000", 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0049999999999999e-05 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.522 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282244+0000", 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011657788000000001 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282277+0000", 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3628999999999999e-05 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.0s1 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:48:55.523 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271300+0000", 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.162050335, 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015401247, 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271300+0000", 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271300+0000", 2024-09-15T02:48:55.524 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271301+0000", 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6800000000000002e-07 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271397+0000", 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6458000000000004e-05 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.525 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271550+0000", 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015337400000000001 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271559+0000", 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6729999999999998e-06 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271758+0000", 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019919 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.526 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271768+0000", 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9490000000000008e-06 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.286630+0000", 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014861846999999999 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.286702+0000", 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1488000000000004e-05 2024-09-15T02:48:55.527 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.3s4 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.275557+0000", 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.157793399, 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0097350240000000001, 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.528 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275557+0000", 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275557+0000", 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275558+0000", 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7199999999999999e-07 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.529 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276687+0000", 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001129307 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276744+0000", 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7287999999999998e-05 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276755+0000", 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0929e-05 2024-09-15T02:48:55.530 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276858+0000", 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010251099999999999 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276877+0000", 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9429999999999999e-05 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285255+0000", 2024-09-15T02:48:55.531 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083775480000000003 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285292+0000", 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7438999999999998e-05 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.0s1 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.276290+0000", 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.1570602400000001, 2024-09-15T02:48:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011969136999999999, 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276290+0000", 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276290+0000", 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.533 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276292+0000", 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.477e-06 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276503+0000", 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000210862 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276563+0000", 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0535000000000002e-05 2024-09-15T02:48:55.534 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276571+0000", 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4930000000000001e-06 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.280195+0000", 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003624118 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.280209+0000", 2024-09-15T02:48:55.535 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4027e-05 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288234+0000", 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0080248479999999994 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288260+0000", 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5777000000000001e-05 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.536 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.537 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.537 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:55.538 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:48:55.539 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:55.531+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:55.641 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:55.637+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:55.646 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:55.646 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:55.646 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:55.646 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:55.655 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:48:55.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:48:55.741 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:55.737+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:55.753 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:55.753 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:55.754 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:55.754 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:55.754 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:55.763 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:48:55.842 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:55.837+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:55.864 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:55.864 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.577346+0000", 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.282897932000001, 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019179986, 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.865 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577346+0000", 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577346+0000", 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577347+0000", 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.35e-07 2024-09-15T02:48:55.866 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577748+0000", 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00040174200000000002 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577791+0000", 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2456999999999997e-05 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577804+0000", 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577967+0000", 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016357899999999999 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.577985+0000", 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7880000000000002e-05 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.868 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596468+0000", 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018483133999999998 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596526+0000", 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7969e-05 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.870 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=100, reqid=client.4526.0:245, at_version=40'29, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:55.870 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.588588+0000", 2024-09-15T02:48:55.870 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.271656498, 2024-09-15T02:48:55.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087388440000000008, 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588588+0000", 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588588+0000", 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588590+0000", 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3149999999999999e-06 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588712+0000", 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012221000000000001 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588784+0000", 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1898999999999997e-05 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.588795+0000", 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0878e-05 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.590216+0000", 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001420803 2024-09-15T02:48:55.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.590233+0000", 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6875e-05 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.597294+0000", 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070608559999999999 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.597327+0000", 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3008e-05 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 40/33 ECSubWrite(tid=101, reqid=client.4526.0:246, at_version=40'30, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:44.592222+0000", 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.268022381, 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0073089950000000004, 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592222+0000", 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592222+0000", 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592224+0000", 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2139999999999999e-06 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592261+0000", 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6436000000000001e-05 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592328+0000", 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7535999999999998e-05 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.592348+0000", 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0080000000000001e-05 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596634+0000", 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042860229999999999 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.596664+0000", 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9774e-05 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.599492+0000", 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028282899999999998 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:44.599531+0000", 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8642e-05 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.as5 44/33 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=44'15, trim_to=0'0, roll_forward_to=43'13))", 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 7.2499946299999998, 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070532809999999998, 2024-09-15T02:48:55.880 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610250+0000", 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.17e-07 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610273+0000", 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3404000000000002e-05 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610322+0000", 2024-09-15T02:48:55.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8609000000000002e-05 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610329+0000", 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8600000000000004e-06 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610427+0000", 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8215999999999996e-05 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.610447+0000", 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0149999999999999e-05 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617264+0000", 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0068173050000000001 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.617303+0000", 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8519999999999997e-05 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.884 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:49.633286+0000", 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.2269583339999999, 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016310313999999999, 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.633286+0000", 2024-09-15T02:48:55.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.633286+0000", 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.633288+0000", 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.917e-06 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.633927+0000", 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063882200000000004 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634033+0000", 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105987 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634061+0000", 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8464000000000001e-05 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634308+0000", 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024650699999999998 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.634331+0000", 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3116999999999999e-05 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.649565+0000", 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015234533999999999 2024-09-15T02:48:55.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.649596+0000", 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0966000000000001e-05 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/33 ECSubWrite(tid=155, reqid=client.4526.0:355, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.675653+0000", 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1845916289999998, 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016585936999999999, 2024-09-15T02:48:55.889 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.675653+0000", 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.675653+0000", 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.675653+0000", 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7300000000000002e-07 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.675964+0000", 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031069000000000002 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.676056+0000", 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1843999999999997e-05 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.676075+0000", 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.942e-05 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.676243+0000", 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016766599999999999 2024-09-15T02:48:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.676294+0000", 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1847e-05 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692192+0000", 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015897826 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692239+0000", 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6270999999999998e-05 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/33 ECSubWrite(tid=156, reqid=client.4526.0:356, at_version=49'19, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.682556+0000", 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1776880580000002, 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012105194, 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.682556+0000", 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.682556+0000", 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.682557+0000", 2024-09-15T02:48:55.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5600000000000004e-07 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.682954+0000", 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00039639999999999999 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.683008+0000", 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4877999999999999e-05 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.683016+0000", 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9740000000000002e-06 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.686517+0000", 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0035005069999999999 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.686535+0000", 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8498000000000001e-05 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694636+0000", 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0081002660000000001 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694661+0000", 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5715e-05 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.898 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/33 ECSubWrite(tid=157, reqid=client.4526.0:357, at_version=49'20, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.690505+0000", 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1697391430000001, 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0084692659999999996, 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.690505+0000", 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.690505+0000", 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.690506+0000", 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0300000000000001e-06 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.690576+0000", 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9949000000000004e-05 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.690612+0000", 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5729000000000003e-05 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.690617+0000", 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5049999999999998e-06 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692301+0000", 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016835719999999999 2024-09-15T02:48:55.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692316+0000", 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5392000000000001e-05 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.698941+0000", 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0066246589999999998 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.698974+0000", 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3429999999999997e-05 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/33 ECSubWrite(tid=159, reqid=client.4526.0:358, at_version=49'21, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.692417+0000", 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1678272449999998, 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0065897059999999999, 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692417+0000", 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692417+0000", 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692418+0000", 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9399999999999993e-07 2024-09-15T02:48:55.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692445+0000", 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6534e-05 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692496+0000", 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1356000000000001e-05 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.692504+0000", 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9489999999999996e-06 2024-09-15T02:48:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694414+0000", 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0019100549999999999 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694427+0000", 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2838e-05 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.698979+0000", 2024-09-15T02:48:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045521650000000004 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.699007+0000", 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7815000000000001e-05 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.as5 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.693939+0000", 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1663053209999998, 2024-09-15T02:48:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0067520619999999996, 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.693939+0000", 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.693939+0000", 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.693939+0000", 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2599999999999998e-07 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694410+0000", 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00047085600000000002 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694446+0000", 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5753000000000001e-05 2024-09-15T02:48:55.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694454+0000", 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2509999999999994e-06 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694532+0000", 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8143e-05 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.694539+0000", 2024-09-15T02:48:55.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7780000000000002e-06 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.700649+0000", 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.006109443 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.700691+0000", 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2412e-05 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.911 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.748872+0000", 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1113724060000001, 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087287979999999994, 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.748872+0000", 2024-09-15T02:48:55.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.748872+0000", 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.748873+0000", 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.308e-06 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.749067+0000", 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019391300000000001 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.749125+0000", 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8023000000000003e-05 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.749146+0000", 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0933000000000001e-05 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.749234+0000", 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7910000000000004e-05 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.749250+0000", 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5939000000000001e-05 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757549+0000", 2024-09-15T02:48:55.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0082995819999999998 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757601+0000", 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1190000000000003e-05 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.es5 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.759664+0000", 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.1005800030000001, 2024-09-15T02:48:55.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016530763, 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759664+0000", 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759664+0000", 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759665+0000", 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6500000000000008e-07 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759799+0000", 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013383399999999999 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759893+0000", 2024-09-15T02:48:55.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3933999999999999e-05 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759907+0000", 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3824e-05 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.760030+0000", 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012286900000000001 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.760052+0000", 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2140000000000001e-05 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776166+0000", 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016114278999999999 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776195+0000", 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8918e-05 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.920 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.es5 49/33 ECSubWrite(tid=48, reqid=client.4526.0:404, at_version=49'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.769805+0000", 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0904396009999999, 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007107693, 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769805+0000", 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769805+0000", 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769806+0000", 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.612e-06 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769913+0000", 2024-09-15T02:48:55.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000106544 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769970+0000", 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7336000000000001e-05 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769977+0000", 2024-09-15T02:48:55.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1980000000000004e-06 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.770987+0000", 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010091779999999999 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.770998+0000", 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1208e-05 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776852+0000", 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058546910000000004 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776912+0000", 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9926000000000002e-05 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.925 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.7s1 49/33 ECSubRead(tid=187, to_read={3:ee3b3dd3:::smithi19125844-46:head=0,4096,0}, subchunks={3:ee3b3dd3:::smithi19125844-46:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.776151+0000", 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0840934120000001, 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17914055500000001, 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776151+0000", 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776151+0000", 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776152+0000", 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7800000000000001e-07 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.776161+0000", 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2e-06 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954759+0000", 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17859862000000001 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954773+0000", 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.365e-05 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954994+0000", 2024-09-15T02:48:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000221277 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955291+0000", 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00029702999999999999 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 49/33 ECSubWrite(tid=188, reqid=client.4526.0:389, at_version=49'57, trim_to=0'0, roll_forward_to=49'55))", 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.954632+0000", 2024-09-15T02:48:55.929 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.9056127140000001, 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.275672959, 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954632+0000", 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954632+0000", 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954633+0000", 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.68e-06 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954712+0000", 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8636999999999999e-05 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.224813+0000", 2024-09-15T02:48:55.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.27010157200000001 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.224831+0000", 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7683999999999999e-05 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.224948+0000", 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000116803 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.224969+0000", 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1103000000000001e-05 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.230250+0000", 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052809600000000003 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.230305+0000", 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.452e-05 2024-09-15T02:48:55.933 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/33 ECSubWrite(tid=189, reqid=client.4526.0:389, at_version=0'0, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.231779+0000", 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.628464833, 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.036909788999999998, 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.231779+0000", 2024-09-15T02:48:55.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.231779+0000", 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.231781+0000", 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6700000000000001e-06 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.231806+0000", 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4545999999999998e-05 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.267107+0000", 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.035301066999999998 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.267113+0000", 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3659999999999997e-06 2024-09-15T02:48:55.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.267184+0000", 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0934999999999998e-05 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.267195+0000", 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0855e-05 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268659+0000", 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0014645439999999999 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268689+0000", 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9805999999999999e-05 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.7s1 50/33 ECSubRead(tid=194, to_read={3:e10d9d4d:::smithi19125844-8:head=0,49152,0}, subchunks={3:e10d9d4d:::smithi19125844-8:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270202+0000", 2024-09-15T02:48:55.938 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.5900426919999999, 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013112209999999999, 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270202+0000", 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270202+0000", 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270202+0000", 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2099999999999995e-07 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270216+0000", 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3573e-05 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:48:55.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012881864999999999 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.283110+0000", 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1936000000000001e-05 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.283197+0000", 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7525e-05 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.283314+0000", 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00011639 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:48:55.942 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.280021+0000", 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.580223656, 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013592053999999999, 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.280021+0000", 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.280021+0000", 2024-09-15T02:48:55.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.280022+0000", 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3999999999999999e-06 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.283062+0000", 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030402010000000002 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288603+0000", 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055410920000000001 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288609+0000", 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3959999999999997e-06 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288655+0000", 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6326999999999999e-05 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288665+0000", 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0045e-05 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293590+0000", 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004925352 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293613+0000", 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2240999999999999e-05 2024-09-15T02:48:55.946 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.288412+0000", 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.5718327479999998, 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013152498, 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288412+0000", 2024-09-15T02:48:55.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288412+0000", 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288412+0000", 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2099999999999995e-07 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.288585+0000", 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017240699999999999 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.296001+0000", 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074162289999999999 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.296006+0000", 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8919999999999999e-06 2024-09-15T02:48:55.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.296070+0000", 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3974999999999996e-05 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.296081+0000", 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0889e-05 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.301543+0000", 2024-09-15T02:48:55.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0054621280000000001 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.301564+0000", 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1056999999999998e-05 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.7s1 50/33 ECSubWrite(tid=204, reqid=client.4526.0:438, at_version=50'64, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.299032+0000", 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.561211812, 2024-09-15T02:48:55.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.25434831099999999, 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299032+0000", 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299032+0000", 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:55.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299033+0000", 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8999999999999996e-07 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299073+0000", 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0225000000000002e-05 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:55.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.550459+0000", 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.25138536099999997 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.550497+0000", 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8449999999999999e-05 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.550652+0000", 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000154637 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.550679+0000", 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.703e-05 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.553329+0000", 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002650398 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.553381+0000", 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1520000000000001e-05 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.955 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.956 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.956 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:55.956 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:55.956 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:55.957 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:48:55.958 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:55.954+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:55.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:48:56.057 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:56.051+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:56.106 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:56.106 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:56.106 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:48:56.106 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:56.115 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:48:56.119 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:56.150 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:56.158 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:56.153+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:56.179 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:56.209 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:56.212 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:56.212 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:48:56.212 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:48:56.212 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:48:56.213 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:56.221 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:48:56.232 INFO:tasks.rados.rados.0.smithi191.stdout:167: done (3 left) 2024-09-15T02:48:56.232 INFO:tasks.rados.rados.0.smithi191.stdout:168: done (2 left) 2024-09-15T02:48:56.232 INFO:tasks.rados.rados.0.smithi191.stdout:169: done (1 left) 2024-09-15T02:48:56.232 INFO:tasks.rados.rados.0.smithi191.stdout:170: done (0 left) 2024-09-15T02:48:56.232 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 43 current snap is 11 2024-09-15T02:48:56.232 INFO:tasks.rados.rados.0.smithi191.stdout:171: seq_num 86 ranges {0=688128,688128=540672,1228800=229376} 2024-09-15T02:48:56.237 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:48:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:56.240 INFO:tasks.rados.rados.0.smithi191.stdout:171: writing smithi19125844-43 from 0 to 688128 tid 1 2024-09-15T02:48:56.245 INFO:tasks.rados.rados.0.smithi191.stdout:171: writing smithi19125844-43 from 688128 to 1228800 tid 2 2024-09-15T02:48:56.248 INFO:tasks.rados.rados.0.smithi191.stdout:171: writing smithi19125844-43 from 1228800 to 1458176 tid 3 2024-09-15T02:48:56.248 INFO:tasks.rados.rados.0.smithi191.stdout:172: rollback oid 14 current snap is 11 2024-09-15T02:48:56.248 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 14 to 5 2024-09-15T02:48:56.248 INFO:tasks.rados.rados.0.smithi191.stdout:173: snap_remove snap 8 2024-09-15T02:48:56.260 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:56.253+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:56.266 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:56.331 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:48:56.331 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:48:56.331 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:237 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.760998754999999, 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013334751000000001, 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 237 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.333 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565422+0000", 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565426+0000", 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2820000000000001e-06 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565442+0000", 2024-09-15T02:48:56.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6096999999999999e-05 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565484+0000", 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2092999999999998e-05 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565500+0000", 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5976000000000001e-05 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565647+0000", 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014730900000000001 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565664+0000", 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6405e-05 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569523+0000", 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003859627 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.569647+0000", 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000123292 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.578757+0000", 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0091106699999999995 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:238 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.760898419, 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016905343999999999, 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.338 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 238 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.339 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565523+0000", 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565524+0000", 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2619999999999999e-06 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565531+0000", 2024-09-15T02:48:56.340 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3300000000000001e-06 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565555+0000", 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3272999999999999e-05 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565567+0000", 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.205e-05 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565812+0000", 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024487000000000001 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565822+0000", 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0356999999999999e-05 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050794789999999996 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571110+0000", 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020819100000000001 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582428+0000", 2024-09-15T02:48:56.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011318531999999999 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:239 3.9s0 3:94f52c52:::smithi19125844-20:head [read 0~327680,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.760824269, 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017215766, 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 239 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.345 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565597+0000", 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565598+0000", 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0669999999999999e-06 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565604+0000", 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7069999999999997e-06 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565633+0000", 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9546000000000001e-05 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565642+0000", 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0950000000000002e-06 2024-09-15T02:48:56.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565870+0000", 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00022745799999999999 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.565885+0000", 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.525e-05 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.348 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571578+0000", 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056929210000000001 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.571650+0000", 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1894e-05 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:43.582813+0000", 2024-09-15T02:48:56.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011162828 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 7.7075664029999995, 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:56.350 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.351 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:48:56.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.353 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.353 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.353 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:48:56.353 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:48:56.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.354 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:303 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:50.615703+0000", 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.7107185469999999, 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014050385, 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.355 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 303 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615703+0000", 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.356 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615703+0000", 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615706+0000", 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9840000000000001e-06 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615720+0000", 2024-09-15T02:48:56.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.399e-05 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615761+0000", 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1709999999999999e-05 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615777+0000", 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5379999999999998e-05 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615928+0000", 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000150803 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615949+0000", 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0993e-05 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.625329+0000", 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093800089999999999 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.625528+0000", 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000199901 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.629753+0000", 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042246150000000001 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:304 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:50.615814+0000", 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.7106075499999998, 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014820155999999999, 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 304 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.361 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615814+0000", 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615814+0000", 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615815+0000", 2024-09-15T02:48:56.362 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.283e-06 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615822+0000", 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.537e-06 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615841+0000", 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8110000000000001e-05 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.615851+0000", 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0257e-05 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.616120+0000", 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00026911399999999999 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.616147+0000", 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7427e-05 2024-09-15T02:48:56.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.626468+0000", 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010320945999999999 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.626662+0000", 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000193437 2024-09-15T02:48:56.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.630634+0000", 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0039720449999999996 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:317 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:50.650444+0000", 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.675977606, 2024-09-15T02:48:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0099475050000000006, 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 317 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650444+0000", 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650444+0000", 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650444+0000", 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1800000000000005e-07 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650449+0000", 2024-09-15T02:48:56.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6739999999999996e-06 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650461+0000", 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2296000000000001e-05 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650470+0000", 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9299999999999992e-06 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.651177+0000", 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000706425 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.651316+0000", 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000138801 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.660391+0000", 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0090755609999999994 2024-09-15T02:48:56.370 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:318 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:50.650478+0000", 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.6759432140000001, 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011383133, 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.371 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 318 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650478+0000", 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650478+0000", 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650479+0000", 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4399999999999999e-07 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650484+0000", 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1420000000000001e-06 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650497+0000", 2024-09-15T02:48:56.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3023999999999999e-05 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.650505+0000", 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.622e-06 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.652330+0000", 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001825208 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.374 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.652493+0000", 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016328800000000001 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:50.661861+0000", 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093681049999999998 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.375 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:416 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~638976] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0653579139999998, 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026381341999999999, 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 416 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.376 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261066+0000", 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1159999999999999e-06 2024-09-15T02:48:56.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262194+0000", 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001127783 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262245+0000", 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0880000000000001e-05 2024-09-15T02:48:56.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262260+0000", 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5065999999999999e-05 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262382+0000", 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012151599999999999 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:56.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262403+0000", 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1344000000000001e-05 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266706+0000", 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004303355 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266794+0000", 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7717000000000001e-05 2024-09-15T02:48:56.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268260+0000", 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001466406 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285160+0000", 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016899773 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287372+0000", 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022116140000000002 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287445+0000", 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2772000000000001e-05 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:417 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.055963964, 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028569959999999998, 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 417 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270459+0000", 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.035e-06 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271813+0000", 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001354037 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271834+0000", 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0302e-05 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271844+0000", 2024-09-15T02:48:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9070000000000002e-06 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274939+0000", 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003095169 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275041+0000", 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000102045 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276123+0000", 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010819880000000001 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285162+0000", 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009039432 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.298984+0000", 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013821599 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299027+0000", 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3445999999999997e-05 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:418 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~147456] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0545614630000002, 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027304571999999999, 2024-09-15T02:48:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 418 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271861+0000", 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1829999999999999e-06 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272037+0000", 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017563799999999999 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272050+0000", 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3506000000000001e-05 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272057+0000", 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999998e-06 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.392 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.392 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281725+0000", 2024-09-15T02:48:56.392 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096680910000000002 2024-09-15T02:48:56.392 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.392 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.392 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281887+0000", 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016169599999999999 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282772+0000", 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088466100000000002 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285439+0000", 2024-09-15T02:48:56.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026669340000000001 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299113+0000", 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013674009000000001 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299164+0000", 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1573999999999998e-05 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:419 3.0s0 3:038350ba:::smithi19125844-11:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0543561659999998, 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.031729234000000002, 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.395 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 419 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:48:56.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272066+0000", 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9000000000000005e-07 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272070+0000", 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1740000000000002e-06 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272078+0000", 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5329999999999997e-06 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272083+0000", 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7859999999999996e-06 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284520+0000", 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012437190000000001 2024-09-15T02:48:56.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284611+0000", 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0816000000000006e-05 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284926+0000", 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031568500000000001 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287991+0000", 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030649599999999998 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303704+0000", 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015713142999999999 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303794+0000", 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9957000000000006e-05 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.401 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:420 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.054333159, 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033404771999999999, 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.402 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 420 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272089+0000", 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.51e-07 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272091+0000", 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1730000000000002e-06 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272096+0000", 2024-09-15T02:48:56.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.904e-06 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272099+0000", 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2059999999999999e-06 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285286+0000", 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013187488000000001 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:48:56.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285374+0000", 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7392999999999995e-05 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304237+0000", 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018863347999999999 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304386+0000", 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014841 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.305493+0000", 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001107299 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:429 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0542598120000002, 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022788739999999998, 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:48:56.407 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 429 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:48:56.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272162+0000", 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272166+0000", 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9269999999999998e-06 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272174+0000", 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6569999999999997e-06 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272178+0000", 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6260000000000003e-06 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274127+0000", 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001949295 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274178+0000", 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0547999999999998e-05 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.294950+0000", 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020772112999999998 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 50/33 ECSubReadReply(tid=60, attrs_read=0))", 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 2.0489865439999999, 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017580023, 2024-09-15T02:48:56.412 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0100000000000004e-07 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293753+0000", 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016317505999999999 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293859+0000", 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105707 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293862+0000", 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4819999999999999e-06 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293910+0000", 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7763000000000003e-05 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.295015+0000", 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011048640000000001 2024-09-15T02:48:56.415 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:443 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~0] snapc 10=[10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:55.230014+0000", 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.096407028, 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013130255, 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.416 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 443 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230014+0000", 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230014+0000", 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230018+0000", 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5499999999999999e-06 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230032+0000", 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.469e-05 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230076+0000", 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3346999999999997e-05 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230097+0000", 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1319999999999999e-05 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230183+0000", 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5871000000000005e-05 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230193+0000", 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0013000000000001e-05 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.234455+0000", 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042619470000000003 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.234536+0000", 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1197000000000002e-05 2024-09-15T02:48:56.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.235238+0000", 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000701887 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.240221+0000", 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0049827129999999997 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.243073+0000", 2024-09-15T02:48:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002852668 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.243144+0000", 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.1051999999999997e-05 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:444 3.0s0 3:038350ba:::smithi19125844-11:head [rollback 10] snapc 10=[10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:55.230125+0000", 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.096296148, 2024-09-15T02:48:56.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014136750999999999, 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 444 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230125+0000", 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230125+0000", 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230127+0000", 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.95e-06 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230133+0000", 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9229999999999999e-06 2024-09-15T02:48:56.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230149+0000", 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6266999999999999e-05 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230165+0000", 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5333e-05 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230355+0000", 2024-09-15T02:48:56.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019084000000000001 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230375+0000", 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9924000000000001e-05 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.235747+0000", 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053714089999999997 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.426 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.235907+0000", 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016019 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.237091+0000", 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011836489999999999 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.242179+0000", 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050882840000000002 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.244222+0000", 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002043549 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.244262+0000", 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9433e-05 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.428 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:445 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~0] snapc 10=[10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:55.230188+0000", 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.0962334410000001, 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014321235, 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 445 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230188+0000", 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230188+0000", 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230189+0000", 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.556e-06 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230194+0000", 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1399999999999999e-06 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230210+0000", 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5324000000000001e-05 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230220+0000", 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0413e-05 2024-09-15T02:48:56.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230422+0000", 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000201399 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.230428+0000", 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.0310000000000004e-06 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.238018+0000", 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075900469999999999 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.238110+0000", 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2641999999999998e-05 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.239400+0000", 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012899859999999999 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.242183+0000", 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0027824450000000001 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.244414+0000", 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002231144 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:55.244509+0000", 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5107999999999998e-05 2024-09-15T02:48:56.434 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:456 3.9s0 3:9c49cf3d:::smithi19125844-14:head [rollback b] snapc 11=[11,10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:56.247452+0000", 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.078969666999999993, 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017192981, 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.435 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 456 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247452+0000", 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247452+0000", 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.436 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247453+0000", 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3740000000000001e-06 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247459+0000", 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5540000000000003e-06 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247477+0000", 2024-09-15T02:48:56.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7683e-05 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247487+0000", 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6059999999999992e-06 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247676+0000", 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018877100000000001 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247689+0000", 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3645000000000001e-05 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.253565+0000", 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.005875875 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.253675+0000", 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000109943 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254359+0000", 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00068416400000000002 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.256091+0000", 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001731808 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264592+0000", 2024-09-15T02:48:56.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0085011679999999999 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264645+0000", 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2389999999999998e-05 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:457 3.9s0 3:9c49cf3d:::smithi19125844-14:head [append 0~0] snapc 11=[11,10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:48:56.441 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.078882748000000003, 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017317169, 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 457 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.442 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247540+0000", 2024-09-15T02:48:56.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1629999999999999e-06 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247546+0000", 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9320000000000001e-06 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247563+0000", 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7268000000000001e-05 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:48:56.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247573+0000", 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8970000000000003e-06 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247766+0000", 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019318499999999999 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247773+0000", 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8600000000000004e-06 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254774+0000", 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070010150000000002 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254891+0000", 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000117342 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.255604+0000", 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071258400000000005 2024-09-15T02:48:56.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.256332+0000", 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072847400000000001 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264776+0000", 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0084437450000000008 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264856+0000", 2024-09-15T02:48:56.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9703999999999994e-05 2024-09-15T02:48:56.448 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.448 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.448 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.448 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:48:56.448 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:48:56.448 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:48:56.449 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:56.449 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:56.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:48:56.450 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:56.446+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:56.451 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:56.479 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:56.507 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:48:56.536 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:56.550 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:56.545+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:56.559 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:56.559 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:56.563 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:56.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:48:56.592 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:56.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:56.611+0000 7f1d7a59c700 0 -- 172.21.15.6:0/2063745741 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f1d7c191d90 msgr2=0x7f1d7c194170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T02:48:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:56.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15] out_osds: [10] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:48:56.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:48:56.646 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-15T02:48:56.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 10 2024-09-15T02:48:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:56.659 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:56.651+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:48:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:56.672 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:56.673 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:56.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:48:56.751 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:56.743+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:56.784 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:56.784 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:56.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:48:56.852 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:48:56.849+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:48:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:56.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:48:56.953 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:48:56.947+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:48:56.963 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-15T02:48:56.955+0000 7f08e39de700 -1 mon.a@0(leader).osd e52 definitely_dead 0 2024-09-15T02:48:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:48:57.054 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:57.050+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:48:57.155 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:57.149+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:57.225 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.226 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:173: done (2 left) 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:174: setattr oid 22 current snap is 11 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:172: finishing rollback tid 0 to smithi19125844-14 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:172: finishing rollback tid 1 to smithi19125844-14 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:172: finishing rollback tid 2 to smithi19125844-14 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 36 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:171: finishing write tid 1 to smithi19125844-43 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:171: finishing write tid 2 to smithi19125844-43 2024-09-15T02:48:57.252 INFO:tasks.rados.rados.0.smithi191.stdout:171: finishing write tid 3 to smithi19125844-43 2024-09-15T02:48:57.253 INFO:tasks.rados.rados.0.smithi191.stdout:171: finishing write tid 4 to smithi19125844-43 2024-09-15T02:48:57.253 INFO:tasks.rados.rados.0.smithi191.stdout:171: finishing write tid 5 to smithi19125844-43 2024-09-15T02:48:57.253 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 25 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-15T02:48:57.253 INFO:tasks.rados.rados.0.smithi191.stdout:171: left oid 43 (ObjNum 86 snap 11 seq_num 86) 2024-09-15T02:48:57.256 INFO:tasks.rados.rados.0.smithi191.stdout:171: done (2 left) 2024-09-15T02:48:57.256 INFO:tasks.rados.rados.0.smithi191.stdout:172: done (1 left) 2024-09-15T02:48:57.256 INFO:tasks.rados.rados.0.smithi191.stdout:175: delete oid 13 current snap is 11 2024-09-15T02:48:57.257 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:57.251+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.10. 2024-09-15T02:48:57.267 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 22 v 24 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-15T02:48:57.273 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-15T02:48:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:48:57.357 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:57.353+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:48:57.457 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:57.451+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:57.558 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:48:57.554+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:48:57.596 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.596 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:48:57.659 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:57.651+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:57.703 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.703 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:48:57.760 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:57.755+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:57.816 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.816 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.825 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:48:57.861 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:57.857+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:57.935 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:57.935 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:57.945 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:48:57.963 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:57.961+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:58.042 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.043 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.051 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:48:58.063 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:58.057+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:58.149 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.149 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:48:58.164 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:58.162+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:58.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:48:58.266 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:58.261+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:58.278 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.278 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:48:58.366 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:58.361+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:58.394 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.394 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:48:58.443 INFO:tasks.rados.rados.0.smithi191.stdout:174: done (1 left) 2024-09-15T02:48:58.443 INFO:tasks.rados.rados.0.smithi191.stdout:175: done (0 left) 2024-09-15T02:48:58.443 INFO:tasks.rados.rados.0.smithi191.stdout:176: copy_from oid 14 from oid 50 current snap is 11 2024-09-15T02:48:58.443 INFO:tasks.rados.rados.0.smithi191.stdout:177: rmattr oid 10 current snap is 11 2024-09-15T02:48:58.443 INFO:tasks.rados.rados.0.smithi191.stdout:178: snap_create 2024-09-15T02:48:58.448 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 10 v 31 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T02:48:58.467 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:58.462+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:58.468 INFO:tasks.rados.rados.0.smithi191.stdout:176: finishing copy_from racing read to smithi19125844-14 2024-09-15T02:48:58.502 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.503 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:48:58.567 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:48:58.565+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:48:58.611 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.611 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:48:58.669 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:58.665+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:58.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:48:58.717 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.717 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:48:58.769 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:58.766+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:58.825 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:48:58.853 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:48:58.863 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.863 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.870 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:48:58.866+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:48:58.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:48:58.886 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:48:58.915 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:48:58.921 INFO:tasks.rados.rados.0.smithi191.stdout:176: finishing copy_from to smithi19125844-14 2024-09-15T02:48:58.921 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-15T02:48:58.942 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:48:58.967 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:48:58.968 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:48:58.970 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:48:58.971 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:58.965+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:58.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:48:58.998 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:48:59.027 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:48:59.054 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:48:59.072 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:48:59.063+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:48:59.074 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:59.074 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:59.075 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:48:59.075 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:48:59.075 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:59.082 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:48:59.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:48:59.110 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:48:59.143 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:48:59.173 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:48:59.167+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:48:59.182 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:48:59.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:48:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:48:59.223+0000 7f0c25d9b700 0 -- 172.21.15.6:0/3826886681 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f0c30192030 msgr2=0x7f0c30194430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:48:59.234 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:59.234 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4311.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:28.404544+0000", 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.824984129000001, 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033046289999999999, 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4311", 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 5 2024-09-15T02:48:59.235 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404544+0000", 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.236 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404544+0000", 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404547+0000", 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.903e-06 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404564+0000", 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6517000000000001e-05 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404602+0000", 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8355999999999997e-05 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404615+0000", 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2916e-05 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404704+0000", 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9290999999999996e-05 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.404920+0000", 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000215718 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 9", 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.405364+0000", 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044384099999999999 2024-09-15T02:48:59.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.406736+0000", 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013719660000000001 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407694+0000", 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00095846600000000005 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407793+0000", 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8686999999999999e-05 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:28.407849+0000", 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5967999999999997e-05 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4311.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-15T02:48:59.241 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:29.419678+0000", 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.809850525000002, 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0027426759999999999, 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4311", 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 7 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419678+0000", 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419678+0000", 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419681+0000", 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9119999999999998e-06 2024-09-15T02:48:59.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419695+0000", 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4532000000000001e-05 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419719+0000", 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3504000000000001e-05 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419747+0000", 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8011999999999999e-05 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419811+0000", 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3761000000000002e-05 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.245 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.419908+0000", 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7108999999999995e-05 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 9", 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.420081+0000", 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017289900000000001 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-09-15T02:48:59.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.421213+0000", 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001132498 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422332+0000", 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001118513 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422383+0000", 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0998000000000002e-05 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:29.422420+0000", 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7938000000000002e-05 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:231 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.664467375999999, 2024-09-15T02:48:59.248 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012963349000000001, 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 231 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.249 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565064+0000", 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1140000000000002e-06 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565078+0000", 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3900000000000001e-05 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565120+0000", 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2070999999999998e-05 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565135+0000", 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5085999999999999e-05 2024-09-15T02:48:59.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565263+0000", 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012826600000000001 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565272+0000", 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0259999999999996e-06 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569405+0000", 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041323439999999996 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569560+0000", 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015537600000000001 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.570911+0000", 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013509349999999999 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.253 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574145+0000", 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003233685 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.577972+0000", 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038275480000000001 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578024+0000", 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1997999999999999e-05 2024-09-15T02:48:59.254 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:232 3.5s0 3:a906e5df:::smithi19125844-36:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.664363418000001, 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013333868, 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.255 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 232 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.256 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565166+0000", 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.054e-06 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565170+0000", 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1969999999999998e-06 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565205+0000", 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4829999999999997e-05 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565234+0000", 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9320999999999999e-05 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565392+0000", 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015784999999999999 2024-09-15T02:48:59.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565408+0000", 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5434000000000002e-05 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571539+0000", 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061311990000000004 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571658+0000", 2024-09-15T02:48:59.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119486 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572348+0000", 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068956300000000002 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574147+0000", 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017987459999999999 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578458+0000", 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043109690000000004 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578499+0000", 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1219e-05 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.261 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:233 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.664283426000001, 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.49017684700000003, 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 233 2024-09-15T02:48:59.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565246+0000", 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565250+0000", 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.76e-06 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565261+0000", 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1693e-05 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565268+0000", 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3570000000000003e-06 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565481+0000", 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021371699999999999 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565537+0000", 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5838000000000003e-05 2024-09-15T02:48:59.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572781+0000", 2024-09-15T02:48:59.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072440639999999997 2024-09-15T02:48:59.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572928+0000", 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014673400000000001 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.573788+0000", 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00086026199999999998 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582219+0000", 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084309350000000005 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055303+0000", 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.47308344099999999 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055422+0000", 2024-09-15T02:48:59.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119119 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/33 ECSubWrite(tid=22, reqid=client.4526.0:235, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.572688+0000", 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.656839878, 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060776770000000001, 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.270 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572688+0000", 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572688+0000", 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572689+0000", 2024-09-15T02:48:59.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.02e-06 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572714+0000", 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4689999999999999e-05 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572763+0000", 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8417000000000001e-05 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572771+0000", 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0970000000000006e-06 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.573105+0000", 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033470700000000001 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.573118+0000", 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2393e-05 2024-09-15T02:48:59.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578734+0000", 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056163289999999998 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578766+0000", 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2023999999999997e-05 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/33 ECSubWrite(tid=23, reqid=client.4526.0:236, at_version=39'18, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.655457821000001, 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089568029999999993, 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6399999999999998e-07 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574843+0000", 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00077236399999999999 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575043+0000", 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019989399999999999 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575050+0000", 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.528e-06 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575124+0000", 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4041999999999993e-05 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575134+0000", 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582994+0000", 2024-09-15T02:48:59.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078592579999999992 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.583027+0000", 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3747000000000001e-05 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 39/33 ECSubWriteReply(tid=18, last_complete=39'11, committed=1, applied=1))", 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.65327162, 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.480047051, 2024-09-15T02:48:59.279 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576262+0000", 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.074e-06 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055023+0000", 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.478760992 2024-09-15T02:48:59.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055037+0000", 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3607e-05 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055156+0000", 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118522 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.056304+0000", 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001148154 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:249 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~704512] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:48:59.283 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.643888162, 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031968222999999997, 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585643+0000", 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7839999999999999e-06 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587905+0000", 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022617729999999999 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587945+0000", 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.981e-05 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587959+0000", 2024-09-15T02:48:59.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4297000000000001e-05 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588041+0000", 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2514000000000003e-05 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588163+0000", 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012119 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589878+0000", 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017156809999999999 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607051+0000", 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017173134999999999 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617495+0000", 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010443661999999999 2024-09-15T02:48:59.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617608+0000", 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113377 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:250 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~376832] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.640610050999999, 2024-09-15T02:48:59.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028844499999999999, 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 250 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588920+0000", 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.807e-06 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589853+0000", 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00093252000000000005 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589887+0000", 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4721000000000002e-05 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589910+0000", 2024-09-15T02:48:59.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2787000000000001e-05 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601237+0000", 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011326924 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601404+0000", 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016697000000000001 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.602323+0000", 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00091864900000000001 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607163+0000", 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048403409999999997 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617711+0000", 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010548119999999999 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617763+0000", 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1660999999999999e-05 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 44/33 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=44'15, trim_to=0'0, roll_forward_to=43'13))", 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:48.610096+0000", 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.619432615000001, 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060650069999999999, 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.295 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610096+0000", 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610096+0000", 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610096+0000", 2024-09-15T02:48:59.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7899999999999999e-07 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610125+0000", 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8509000000000001e-05 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610205+0000", 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0733000000000001e-05 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610216+0000", 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0812000000000001e-05 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610327+0000", 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011075599999999999 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.610345+0000", 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8363e-05 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.616129+0000", 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057837219999999998 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.616161+0000", 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1733000000000003e-05 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.299 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/33 ECSubWrite(tid=29, reqid=osd.15.0:29, at_version=44'20, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:48.615428+0000", 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.614100767, 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041437999999999996, 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615428+0000", 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615428+0000", 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615429+0000", 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7399999999999991e-07 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615458+0000", 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9643000000000001e-05 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615596+0000", 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013814400000000001 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615605+0000", 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8589999999999994e-06 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615674+0000", 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8455999999999996e-05 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.615692+0000", 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7859999999999998e-05 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.619537+0000", 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038452349999999998 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.619571+0000", 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4629000000000003e-05 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.303 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:307 3.5s0 3:ae24c343:::smithi19125844-3:head [copy-from ver 10] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 8.6102997450000007, 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.021447352999999999, 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.304 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 307 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619232+0000", 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619263+0000", 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1090000000000002e-05 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619289+0000", 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5284000000000001e-05 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619300+0000", 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1518000000000001e-05 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619367+0000", 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6983000000000006e-05 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619450+0000", 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2785000000000001e-05 2024-09-15T02:48:59.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.630087+0000", 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010637047 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635664+0000", 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055774500000000003 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640607+0000", 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049425019999999997 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640676+0000", 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9078000000000005e-05 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:308 3.5s0 3:ae24c343:::smithi19125844-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:48:59.309 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 8.6102101090000005, 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016310021000000001, 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 308 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619319+0000", 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9500000000000002e-07 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619324+0000", 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.87e-06 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619332+0000", 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2539999999999992e-06 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619335+0000", 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0079999999999998e-06 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619592+0000", 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000256724 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619634+0000", 2024-09-15T02:48:59.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2011999999999997e-05 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635446+0000", 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015811935999999999 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635533+0000", 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7126000000000006e-05 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635628+0000", 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5495999999999995e-05 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 46/33 ECSubWrite(tid=131, reqid=client.4526.0:306, at_version=46'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.627774+0000", 2024-09-15T02:48:59.315 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 8.6017546869999997, 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029363140000000002, 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.627774+0000", 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.627774+0000", 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.627774+0000", 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0999999999999999e-07 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.628003+0000", 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022920300000000001 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.628057+0000", 2024-09-15T02:48:59.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3242000000000003e-05 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.628066+0000", 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8260000000000005e-06 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.628147+0000", 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0898999999999999e-05 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.628162+0000", 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4867e-05 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.630677+0000", 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025151570000000001 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.630710+0000", 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2610000000000001e-05 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.319 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 8.5794381350000002, 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013723521000000001, 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:48:59.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650091+0000", 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04e-06 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650742+0000", 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000650436 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650792+0000", 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0680000000000003e-05 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650812+0000", 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9397000000000001e-05 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650905+0000", 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3294000000000005e-05 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650922+0000", 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7209000000000001e-05 2024-09-15T02:48:59.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663784+0000", 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012861490999999999 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663814+0000", 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9974000000000001e-05 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/33 ECSubWrite(tid=164, reqid=client.4526.0:313, at_version=46'24, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.658085+0000", 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 8.5714432200000008, 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059411050000000003, 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.658085+0000", 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.658085+0000", 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.658087+0000", 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4419999999999999e-06 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.658233+0000", 2024-09-15T02:48:59.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014611999999999999 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.658296+0000", 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2978999999999995e-05 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.658325+0000", 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8887999999999999e-05 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.660600+0000", 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002275057 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.660611+0000", 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1260000000000001e-05 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664001+0000", 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003390059 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664026+0000", 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5299999999999998e-05 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 49/33 ECSubWrite(tid=175, reqid=client.4526.0:354, at_version=49'27, trim_to=0'0, roll_forward_to=46'25))", 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.667878+0000", 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.5616503169999998, 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064772570000000002, 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.329 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.667878+0000", 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.667878+0000", 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.667879+0000", 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0299999999999999e-07 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668189+0000", 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000310582 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668318+0000", 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012915100000000001 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668339+0000", 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0990000000000001e-05 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668545+0000", 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000205995 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668575+0000", 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0253999999999999e-05 2024-09-15T02:48:59.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.674297+0000", 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057217830000000003 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.674355+0000", 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7998999999999998e-05 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.5277582130000003, 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084791039999999995, 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5699999999999999e-07 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701896+0000", 2024-09-15T02:48:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012543400000000001 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701967+0000", 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1172000000000002e-05 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701978+0000", 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1e-05 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702056+0000", 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7615999999999997e-05 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702069+0000", 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3746e-05 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710220+0000", 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081502959999999996 2024-09-15T02:48:59.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710249+0000", 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9583000000000001e-05 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.9541110719999999, 2024-09-15T02:48:59.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092230960000000001, 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3300000000000001e-07 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276746+0000", 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013288589999999999 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276893+0000", 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000146849 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276906+0000", 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2347000000000001e-05 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276979+0000", 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3198e-05 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277005+0000", 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5837e-05 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284582+0000", 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075769779999999998 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284640+0000", 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8795000000000002e-05 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.343 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:59.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:48:59.346 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:48:59.347 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:48:59.347 INFO:tasks.rados.rados.0.smithi191.stdout:176: done (2 left) 2024-09-15T02:48:59.347 INFO:tasks.rados.rados.0.smithi191.stdout:177: done (1 left) 2024-09-15T02:48:59.347 INFO:tasks.rados.rados.0.smithi191.stdout:178: done (0 left) 2024-09-15T02:48:59.347 INFO:tasks.rados.rados.0.smithi191.stdout:179: rollback oid 45 current snap is 12 2024-09-15T02:48:59.347 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 45 to 7 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:180: delete oid 23 current snap is 12 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:179: finishing rollback tid 0 to smithi19125844-45 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:179: finishing rollback tid 1 to smithi19125844-45 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 45 v 34 (ObjNum 1361 snap 0 seq_num 1708044320) dirty dne 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:179: done (1 left) 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:180: done (0 left) 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:181: delete oid 47 current snap is 12 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:181: done (0 left) 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:182: read oid 41 snap -1 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:182: expect (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:48:59.348 INFO:tasks.rados.rados.0.smithi191.stdout:183: copy_from oid 28 from oid 20 current snap is 12 2024-09-15T02:48:59.349 INFO:tasks.rados.rados.0.smithi191.stdout:184: snap_create 2024-09-15T02:48:59.349 INFO:tasks.rados.rados.0.smithi191.stdout:182: expect (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:48:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:48:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:48:59.351 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:48:59.341+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:48:59.445 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:48:59.441+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:48:59.497 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:59.497 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:48:59.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.497 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:472 3.as0 3:54c1cad6:::smithi19125844-28:head [copy-from ver 18 in=55b] snapc 13=[13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:48:59.497 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:48:59.497 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.17437765599999999, 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17443176199999999, 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 472 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:48:59.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321107+0000", 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9740000000000001e-06 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321117+0000", 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0214000000000001e-05 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321173+0000", 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5482000000000002e-05 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321185+0000", 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2028000000000001e-05 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321271+0000", 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6592e-05 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321500+0000", 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022896 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.501 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:473 3.as0 3:54c1cad6:::smithi19125844-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.174277878, 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17444848399999999, 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for blocked object", 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 473 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321206+0000", 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2440000000000001e-06 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321211+0000", 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4580000000000001e-06 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321237+0000", 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5111999999999999e-05 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321246+0000", 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1519999999999998e-06 2024-09-15T02:48:59.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321773+0000", 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052693800000000002 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321843+0000", 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0831e-05 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.505 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-15T02:48:59.506 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 2 2024-09-15T02:48:59.506 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:59.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:48:59.546 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:48:59.539+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:48:59.607 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:59.608 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:48:59.608 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:48:59.608 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:48:59.608 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:59.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:48:59.647 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:48:59.641+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:48:59.748 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:48:59.741+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:48:59.766 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:48:59.766 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.568653+0000", 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.194747080999999, 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013097361, 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.767 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568653+0000", 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568653+0000", 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568654+0000", 2024-09-15T02:48:59.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2399999999999996e-07 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569309+0000", 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065497999999999999 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569378+0000", 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9105999999999999e-05 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569390+0000", 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.171e-05 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569510+0000", 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012006399999999999 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569534+0000", 2024-09-15T02:48:59.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4337000000000001e-05 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.581715+0000", 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012180471999999999 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.581750+0000", 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5768e-05 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:241 3.as0 3:517ce0e6:::smithi19125844-47:head [read 0~786432,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:44.554008+0000", 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.209392492999999, 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084581350000000003, 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.772 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554008+0000", 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554008+0000", 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554012+0000", 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3100000000000002e-06 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554028+0000", 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6263e-05 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554063+0000", 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5311000000000001e-05 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554077+0000", 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3803e-05 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554231+0000", 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015404899999999999 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554548+0000", 2024-09-15T02:48:59.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00031640900000000003 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.562466+0000", 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079179899999999998 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:242 3.as0 3:517ce0e6:::smithi19125844-47:head [read 0~786432,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:44.554099+0000", 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.209300939, 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096035200000000008, 2024-09-15T02:48:59.776 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 242 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554099+0000", 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554099+0000", 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554100+0000", 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1570000000000001e-06 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554108+0000", 2024-09-15T02:48:59.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3329999999999999e-06 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554118+0000", 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.047e-05 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554140+0000", 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1620999999999998e-05 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.779 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.556617+0000", 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024775320000000002 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.556742+0000", 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012471499999999999 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.563703+0000", 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069606920000000001 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.780 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:243 3.as0 3:517ce0e6:::smithi19125844-47:head [read 0~786432,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:44.554204+0000", 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.209196467, 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010835216, 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.781 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 243 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554204+0000", 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554204+0000", 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554205+0000", 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.24e-06 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554213+0000", 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7300000000000005e-06 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554234+0000", 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1234e-05 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.554245+0000", 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1586000000000001e-05 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.557487+0000", 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032420729999999998 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.557770+0000", 2024-09-15T02:48:59.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000282873 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.565039+0000", 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072684799999999999 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/33 ECSubWrite(tid=119, reqid=osd.13.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:48.611889+0000", 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.151511148000001, 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043096200000000001, 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611889+0000", 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611889+0000", 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611890+0000", 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1699999999999998e-07 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611901+0000", 2024-09-15T02:48:59.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1885999999999999e-05 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611938+0000", 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6826000000000003e-05 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611947+0000", 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2640000000000008e-06 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.788 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611988+0000", 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1570999999999999e-05 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.611994+0000", 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0789999999999997e-06 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.616165+0000", 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041705170000000003 2024-09-15T02:48:59.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:48.616199+0000", 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3859999999999998e-05 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:360 3.as0 3:54c1cad6:::smithi19125844-28:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0874162739999997, 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034523553999999998, 2024-09-15T02:48:59.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 360 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.791 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675987+0000", 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0900000000000001e-06 2024-09-15T02:48:59.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676003+0000", 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6407000000000001e-05 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676045+0000", 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1390000000000002e-05 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676065+0000", 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0157e-05 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676146+0000", 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0891999999999997e-05 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676153+0000", 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0079999999999996e-06 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.678914+0000", 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002761262 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679005+0000", 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0648000000000004e-05 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.693685+0000", 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014680723 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702650+0000", 2024-09-15T02:48:59.796 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089650400000000005 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710453+0000", 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078028220000000001 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710507+0000", 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4114999999999999e-05 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.797 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:361 3.as0 3:54c1cad6:::smithi19125844-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0873026929999998, 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026448546, 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 361 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.798 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676099+0000", 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.72e-06 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676105+0000", 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3729999999999998e-06 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676128+0000", 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2096999999999999e-05 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676139+0000", 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1603999999999999e-05 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676214+0000", 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4380000000000003e-05 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676221+0000", 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8970000000000008e-06 2024-09-15T02:48:59.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679152+0000", 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029307930000000001 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679218+0000", 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5701e-05 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702293+0000", 2024-09-15T02:48:59.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.02307476 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702424+0000", 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013107900000000001 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702546+0000", 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000122142 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(osd.12.22:1 3.ds0 3:bcf25562:::smithi19125844-42:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.683926+0000", 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0794742920000004, 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066395890000000004, 2024-09-15T02:48:59.804 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.683926+0000", 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.683926+0000", 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.683927+0000", 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.68e-06 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.683934+0000", 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3679999999999998e-06 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.806 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.683955+0000", 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0797000000000001e-05 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.683962+0000", 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2520000000000002e-06 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.684053+0000", 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1063999999999994e-05 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.684202+0000", 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014880699999999999 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.690565+0000", 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063636209999999999 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:48:59.808 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.691753+0000", 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0716465849999999, 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084425750000000008, 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691753+0000", 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691753+0000", 2024-09-15T02:48:59.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691754+0000", 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1399999999999998e-07 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692274+0000", 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051977600000000001 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692337+0000", 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3224000000000003e-05 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692352+0000", 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5104e-05 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692421+0000", 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9509000000000001e-05 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692432+0000", 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0234000000000001e-05 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.700164+0000", 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0077321660000000004 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.700196+0000", 2024-09-15T02:48:59.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2248e-05 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/33 ECSubWrite(tid=160, reqid=client.4526.0:364, at_version=49'23, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.701292+0000", 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0621082389999996, 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0099239840000000003, 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701292+0000", 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701292+0000", 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.814 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701293+0000", 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7000000000000004e-07 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701362+0000", 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9874000000000002e-05 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701727+0000", 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036467100000000001 2024-09-15T02:48:59.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701790+0000", 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3305000000000005e-05 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701860+0000", 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9406000000000006e-05 2024-09-15T02:48:59.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701874+0000", 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4100000000000001e-05 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711189+0000", 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0093150219999999992 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711216+0000", 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6835999999999999e-05 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/33 ECSubWrite(tid=161, reqid=client.4526.0:365, at_version=49'24, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.702065+0000", 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0613347360000001, 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095835980000000005, 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.818 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702065+0000", 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702065+0000", 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702066+0000", 2024-09-15T02:48:59.819 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1060000000000001e-06 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702136+0000", 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9492999999999993e-05 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702197+0000", 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0921000000000002e-05 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702204+0000", 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5390000000000002e-06 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708915+0000", 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067110420000000004 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708931+0000", 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5702999999999999e-05 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711621+0000", 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002690202 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711649+0000", 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7591999999999999e-05 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/33 ECSubWrite(tid=164, reqid=client.4526.0:366, at_version=49'25, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.704337+0000", 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0590632170000003, 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073239289999999999, 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.704337+0000", 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.823 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.704337+0000", 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.704337+0000", 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2799999999999996e-07 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.704352+0000", 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5082999999999999e-05 2024-09-15T02:48:59.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.704428+0000", 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5543999999999994e-05 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.704451+0000", 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2591000000000001e-05 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710997+0000", 2024-09-15T02:48:59.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065463739999999998 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711008+0000", 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1051e-05 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711652+0000", 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00064441300000000004 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.711661+0000", 2024-09-15T02:48:59.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3450000000000006e-06 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/33 ECSubWrite(tid=170, reqid=client.4526.0:379, at_version=49'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.718338+0000", 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.0450623219999997, 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065097489999999996, 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718338+0000", 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718338+0000", 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718338+0000", 2024-09-15T02:48:59.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9800000000000003e-07 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718618+0000", 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027946600000000002 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718657+0000", 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8977000000000003e-05 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718666+0000", 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8349999999999993e-06 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718730+0000", 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4264000000000007e-05 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.718746+0000", 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5959000000000001e-05 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.724809+0000", 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060631629999999999 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.724848+0000", 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8386999999999997e-05 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/33 ECSubWrite(tid=171, reqid=client.4526.0:380, at_version=49'28, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.724967+0000", 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.038433017, 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076738409999999998, 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.724967+0000", 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.724967+0000", 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.724968+0000", 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6400000000000002e-07 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.725316+0000", 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00034836300000000003 2024-09-15T02:48:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.725350+0000", 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.392e-05 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.725356+0000", 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5770000000000001e-06 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.725393+0000", 2024-09-15T02:48:59.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6919999999999999e-05 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.725398+0000", 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9819999999999996e-06 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732586+0000", 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0071872009999999998 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732641+0000", 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5213999999999997e-05 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.5095364399999998, 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.02916968, 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.836 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254277+0000", 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00041306699999999998 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254344+0000", 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6806999999999999e-05 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254359+0000", 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4738e-05 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254477+0000", 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118509 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254495+0000", 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7297999999999999e-05 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.282999+0000", 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028504345 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283033+0000", 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4436e-05 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.5028145850000003, 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022480231, 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260587+0000", 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7400000000000001e-06 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260827+0000", 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000239459 2024-09-15T02:48:59.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260907+0000", 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9956999999999994e-05 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260923+0000", 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6592e-05 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268606+0000", 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076824390000000001 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268615+0000", 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9600000000000006e-06 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283052+0000", 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014437815 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283066+0000", 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3269e-05 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=195, reqid=client.4526.0:411, at_version=50'34, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.270958+0000", 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.4924417979999998, 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012122575, 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.845 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270958+0000", 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270958+0000", 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270959+0000", 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0139999999999999e-06 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.271081+0000", 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012218600000000001 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.271126+0000", 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4116000000000002e-05 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.271133+0000", 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4089999999999997e-06 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275065+0000", 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039322519999999998 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275097+0000", 2024-09-15T02:48:59.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2125000000000002e-05 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283070+0000", 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079729540000000008 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283081+0000", 2024-09-15T02:48:59.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0519e-05 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s6 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275529+0000", 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.4878707650000003, 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011397446, 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275529+0000", 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275529+0000", 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275530+0000", 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0499999999999999e-07 2024-09-15T02:48:59.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276493+0000", 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00096330700000000005 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276670+0000", 2024-09-15T02:48:59.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017730099999999999 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276677+0000", 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1300000000000003e-06 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276782+0000", 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010467900000000001 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0854e-05 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286894+0000", 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010100573 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286927+0000", 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3197000000000001e-05 2024-09-15T02:48:59.854 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=196, reqid=client.4526.0:412, at_version=50'35, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.276132+0000", 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.4872676360000003, 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069595359999999997, 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276132+0000", 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276132+0000", 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276133+0000", 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4e-07 2024-09-15T02:48:59.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276171+0000", 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7842999999999997e-05 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276247+0000", 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5677000000000001e-05 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276257+0000", 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0083e-05 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.281320+0000", 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050631950000000004 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.281338+0000", 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7768999999999999e-05 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283083+0000", 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001745569 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283092+0000", 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5599999999999994e-06 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:468 3.as0 3:517ce0e6:::smithi19125844-47:head [delete] snapc 13=[13,11,10,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e55)", 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.314350+0000", 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.44904980700000002, 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061595870000000002, 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 468 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314350+0000", 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314350+0000", 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:48:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314353+0000", 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.892e-06 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314366+0000", 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2563e-05 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:48:59.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314398+0000", 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1888999999999999e-05 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314410+0000", 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2136e-05 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314540+0000", 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000130524 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.314769+0000", 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0002288 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.316600+0000", 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00183073 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.318580+0000", 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00197978 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.320426+0000", 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001845923 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.320510+0000", 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4350000000000004e-05 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:48:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:48:59.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:48:59.868 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:48:59.862+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:48:59.938 INFO:tasks.rados.rados.0.smithi191.stdout:183: finishing copy_from racing read to smithi19125844-28 2024-09-15T02:48:59.939 INFO:tasks.rados.rados.0.smithi191.stdout:183: finishing copy_from to smithi19125844-28 2024-09-15T02:48:59.939 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 18 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T02:48:59.968 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:48:59.961+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:48:59.978 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:48:59.978 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:48:59.978 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:48:59.978 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:48:59.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:00.069 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:00.066+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:00.093 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.093 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:00.093 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:00.093 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:00.093 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:00.170 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:00.163+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.619968029000001, 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010162803999999999, 2024-09-15T02:49:00.201 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577825+0000", 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9299999999999996e-07 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578183+0000", 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035738600000000001 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578258+0000", 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5891999999999998e-05 2024-09-15T02:49:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578278+0000", 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9641999999999999e-05 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578517+0000", 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000238619 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578582+0000", 2024-09-15T02:49:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5091000000000004e-05 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.587957+0000", 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0093750499999999994 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.587987+0000", 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0230999999999999e-05 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.56365853, 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016823131000000002, 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634136+0000", 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7940000000000001e-06 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634475+0000", 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033968099999999999 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634663+0000", 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018762999999999999 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634681+0000", 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7974000000000001e-05 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634905+0000", 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022468600000000001 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634923+0000", 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7254000000000001e-05 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650923+0000", 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015999777999999999 2024-09-15T02:49:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650957+0000", 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4334e-05 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/33 ECSubWrite(tid=125, reqid=client.4526.0:295, at_version=45'45, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.645691+0000", 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.552100878999999, 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060612399999999999, 2024-09-15T02:49:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645691+0000", 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645691+0000", 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645693+0000", 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8929999999999999e-06 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645864+0000", 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170376 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645985+0000", 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121002 2024-09-15T02:49:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.646005+0000", 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0015000000000001e-05 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647020+0000", 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001015127 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647037+0000", 2024-09-15T02:49:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6748e-05 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651728+0000", 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046918380000000003 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651753+0000", 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4241000000000001e-05 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.5663503779999992, 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082044130000000007, 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.27e-07 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631889+0000", 2024-09-15T02:49:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044670300000000001 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631941+0000", 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1823999999999997e-05 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631956+0000", 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5143999999999999e-05 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632032+0000", 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6149999999999994e-05 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632044+0000", 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2282e-05 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.639605+0000", 2024-09-15T02:49:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075610620000000003 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.639646+0000", 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0821000000000001e-05 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 49/33 ECSubWrite(tid=155, reqid=client.4526.0:355, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.522584406, 2024-09-15T02:49:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019850428999999999, 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9999999999999997e-07 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675436+0000", 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227369 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675628+0000", 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019193300000000001 2024-09-15T02:49:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675646+0000", 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7830999999999999e-05 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.676067+0000", 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042108899999999998 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.676096+0000", 2024-09-15T02:49:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9149000000000002e-05 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695024+0000", 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018927979000000001 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695058+0000", 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4479e-05 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 49/33 ECSubWrite(tid=156, reqid=client.4526.0:356, at_version=49'19, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.5142438890000003, 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011525719, 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683550+0000", 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.03e-06 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684004+0000", 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045311 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684167+0000", 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163869 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684188+0000", 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0190000000000002e-05 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684753+0000", 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056561700000000003 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684768+0000", 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4807000000000001e-05 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695067+0000", 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010299341 2024-09-15T02:49:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695074+0000", 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7549999999999997e-06 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 49/33 ECSubWrite(tid=177, reqid=client.4526.0:371, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.705944+0000", 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.4918486550000001, 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059328430000000001, 2024-09-15T02:49:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.705944+0000", 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.705944+0000", 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.705944+0000", 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.58e-07 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.706301+0000", 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035742199999999998 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.706330+0000", 2024-09-15T02:49:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8586e-05 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.706335+0000", 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8590000000000004e-06 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.706380+0000", 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4956000000000001e-05 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.706388+0000", 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5529999999999993e-06 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.711845+0000", 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054566759999999997 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.711876+0000", 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1433000000000002e-05 2024-09-15T02:49:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 49/33 ECSubWrite(tid=178, reqid=client.4526.0:372, at_version=49'19, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.713790+0000", 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.4840024139999999, 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087937559999999998, 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.713790+0000", 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.713790+0000", 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.713791+0000", 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0300000000000001e-06 2024-09-15T02:49:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714120+0000", 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032886599999999998 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714161+0000", 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1090999999999997e-05 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714171+0000", 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0548e-05 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714229+0000", 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7646999999999999e-05 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714239+0000", 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0227999999999999e-05 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.722547+0000", 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083076820000000003 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.722584+0000", 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6664e-05 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.4482014679999997, 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016121677000000001, 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749593+0000", 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.708e-06 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749692+0000", 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9852000000000006e-05 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749755+0000", 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2292999999999995e-05 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749767+0000", 2024-09-15T02:49:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2646000000000001e-05 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749863+0000", 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5872999999999995e-05 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749883+0000", 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0329000000000001e-05 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.765618+0000", 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015734416000000001 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.765712+0000", 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4560000000000003e-05 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:394 3.4s0 3:2d69f33a:::smithi19125844-35:head [read 0~573440,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752714+0000", 2024-09-15T02:49:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.44507808, 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061023869999999999, 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 394 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752714+0000", 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752714+0000", 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752717+0000", 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.385e-06 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752727+0000", 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0625000000000001e-05 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752753+0000", 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5812e-05 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752767+0000", 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3592e-05 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752901+0000", 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013408199999999999 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.753073+0000", 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017227599999999999 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758817+0000", 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057436149999999997 2024-09-15T02:49:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:395 3.4s0 3:2d69f33a:::smithi19125844-35:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.4449915320000004, 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006832917, 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 395 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752802+0000", 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0869999999999999e-06 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752808+0000", 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6250000000000001e-06 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752844+0000", 2024-09-15T02:49:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5979999999999998e-05 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752855+0000", 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0574000000000001e-05 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.754604+0000", 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017494609999999999 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.754741+0000", 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000136193 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759634+0000", 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048929969999999996 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:396 3.4s0 3:2d69f33a:::smithi19125844-35:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752913+0000", 2024-09-15T02:49:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.4448791319999996, 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008566957, 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 396 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752913+0000", 2024-09-15T02:49:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752913+0000", 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752914+0000", 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752921+0000", 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8929999999999999e-06 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752946+0000", 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4474e-05 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752954+0000", 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1880000000000002e-06 2024-09-15T02:49:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.755131+0000", 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002176978 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.755267+0000", 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135907 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761480+0000", 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006213405 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 49/33 ECSubWrite(tid=188, reqid=client.4526.0:389, at_version=49'57, trim_to=0'0, roll_forward_to=49'55))", 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.955674+0000", 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.2421187429999998, 2024-09-15T02:49:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060873159999999997, 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955674+0000", 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955674+0000", 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955676+0000", 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-06 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955821+0000", 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014501999999999999 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955930+0000", 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109701 2024-09-15T02:49:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955950+0000", 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9783e-05 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.956149+0000", 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000198505 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.956178+0000", 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9629999999999999e-05 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.961685+0000", 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055066749999999999 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.961761+0000", 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5962000000000002e-05 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9209989649999999, 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010710503999999999, 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276794+0000", 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277315+0000", 2024-09-15T02:49:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052080900000000001 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277480+0000", 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016516400000000001 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277511+0000", 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0750999999999997e-05 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277623+0000", 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112213 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277635+0000", 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1899000000000001e-05 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287471+0000", 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098365050000000006 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287504+0000", 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2462999999999997e-05 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.907732942, 2024-09-15T02:49:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016610296, 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290061+0000", 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.327e-06 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290153+0000", 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2641999999999998e-05 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290291+0000", 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013733500000000001 2024-09-15T02:49:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290300+0000", 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0459999999999994e-06 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290359+0000", 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9695999999999999e-05 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290374+0000", 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4831000000000001e-05 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306642+0000", 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016267321000000001 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306670+0000", 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8098000000000001e-05 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9076198780000002, 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016516073999999999, 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290173+0000", 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5400000000000002e-07 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290241+0000", 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7681000000000005e-05 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290326+0000", 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5348999999999999e-05 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290330+0000", 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.498e-06 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297821+0000", 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074905040000000003 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297832+0000", 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0905e-05 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306676+0000", 2024-09-15T02:49:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088443059999999997 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306688+0000", 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2377e-05 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=202, reqid=client.4526.0:432, at_version=50'61, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.296969+0000", 2024-09-15T02:49:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9008229500000002, 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097351390000000003, 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.296969+0000", 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.296969+0000", 2024-09-15T02:49:00.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.296970+0000", 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7699999999999992e-07 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297033+0000", 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2920000000000001e-05 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297081+0000", 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7655999999999999e-05 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297088+0000", 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.973e-06 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304585+0000", 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074971949999999999 2024-09-15T02:49:00.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304596+0000", 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306691+0000", 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020950539999999998 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306704+0000", 2024-09-15T02:49:00.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3756e-05 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs5 50/33 ECSubWrite(tid=39, reqid=client.4526.0:423, at_version=0'0, trim_to=0'0, roll_forward_to=50'9))", 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.297010+0000", 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9007818690000002, 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067341830000000004, 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297010+0000", 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297010+0000", 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297012+0000", 2024-09-15T02:49:00.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.184e-06 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297027+0000", 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5418e-05 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297070+0000", 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2843000000000003e-05 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297082+0000", 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297166+0000", 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4474000000000005e-05 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297179+0000", 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3371e-05 2024-09-15T02:49:00.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303705+0000", 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065259749999999998 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303745+0000", 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9178000000000003e-05 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=203, reqid=client.4526.0:433, at_version=50'62, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.298710+0000", 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8990824960000001, 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080079529999999999, 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298710+0000", 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298710+0000", 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298711+0000", 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5600000000000005e-07 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298732+0000", 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1503000000000001e-05 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298772+0000", 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0191000000000002e-05 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298779+0000", 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7989999999999996e-06 2024-09-15T02:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305762+0000", 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069827149999999996 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305771+0000", 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9090000000000006e-06 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306709+0000", 2024-09-15T02:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093796299999999999 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306718+0000", 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1169999999999993e-06 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=204, reqid=client.4526.0:438, at_version=50'64, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.300087+0000", 2024-09-15T02:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8977052849999998, 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072160439999999996, 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300087+0000", 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300087+0000", 2024-09-15T02:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300088+0000", 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5199999999999995e-07 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300115+0000", 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7169999999999999e-05 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300170+0000", 2024-09-15T02:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4527999999999997e-05 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300176+0000", 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5880000000000003e-06 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306001+0000", 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058244480000000003 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306011+0000", 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0394000000000001e-05 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.307269+0000", 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012578120000000001 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.307303+0000", 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4251999999999997e-05 2024-09-15T02:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.294 INFO:tasks.rados.rados.0.smithi191.stdout:182: done (2 left) 2024-09-15T02:49:00.294 INFO:tasks.rados.rados.0.smithi191.stdout:183: done (1 left) 2024-09-15T02:49:00.294 INFO:tasks.rados.rados.0.smithi191.stdout:184: done (0 left) 2024-09-15T02:49:00.295 INFO:tasks.rados.rados.0.smithi191.stdout:185: snap_create 2024-09-15T02:49:00.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:00.296 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:00.289+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:00.395 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:00.389+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:00.401 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.402 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:00.402 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:00.402 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:00.497 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:00.493+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:00.513 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.513 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:00.513 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:00.513 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:00.513 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:00.598 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:00.594+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:00.622 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.622 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:00.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:00.623 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:00.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.623 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:49:00.623 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:00.623 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.040775872000001, 2024-09-15T02:49:00.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014940116999999999, 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577945+0000", 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.037e-06 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:49:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035186899999999999 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578459+0000", 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016206200000000001 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578482+0000", 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3070000000000001e-05 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578643+0000", 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016161999999999999 2024-09-15T02:49:00.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578670+0000", 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592839+0000", 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014168557 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592884+0000", 2024-09-15T02:49:00.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4786000000000001e-05 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/33 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=44'21, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:49:00.631 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.020696832, 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044055149999999996, 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:49:00.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:49:00.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598023+0000", 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.96e-07 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598066+0000", 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2595999999999997e-05 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598773+0000", 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070733500000000002 2024-09-15T02:49:00.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598791+0000", 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8450000000000001e-05 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598890+0000", 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8510000000000004e-05 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598910+0000", 2024-09-15T02:49:00.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0514999999999999e-05 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602080+0000", 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031698780000000001 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602428+0000", 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034783500000000001 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.600217+0000", 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.018502877, 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057404500000000002, 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.640 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600217+0000", 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600217+0000", 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600218+0000", 2024-09-15T02:49:00.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.356e-06 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600491+0000", 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027357800000000002 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600579+0000", 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7216e-05 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600592+0000", 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2798e-05 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600866+0000", 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027486999999999998 2024-09-15T02:49:00.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600893+0000", 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6843e-05 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.605899+0000", 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050058239999999999 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.605957+0000", 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7964999999999998e-05 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.647 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.984513737, 2024-09-15T02:49:00.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016865508000000001, 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:00.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0999999999999999e-07 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634501+0000", 2024-09-15T02:49:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029489499999999999 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634605+0000", 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103488 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634624+0000", 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9530000000000001e-05 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634715+0000", 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1284000000000002e-05 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634741+0000", 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5350999999999998e-05 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651036+0000", 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016295245999999999 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651071+0000", 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5203999999999997e-05 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=125, reqid=client.4526.0:295, at_version=45'45, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.645673+0000", 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.973046700999999, 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061112989999999997, 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645673+0000", 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645673+0000", 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645674+0000", 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0440000000000001e-06 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645765+0000", 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1082999999999995e-05 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645824+0000", 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9073999999999997e-05 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.645838+0000", 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3725e-05 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647211+0000", 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001373193 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.647221+0000", 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0575e-05 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651761+0000", 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004540021 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651784+0000", 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2583999999999999e-05 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.636542+0000", 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.9821772469999992, 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052986020000000003, 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636542+0000", 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636542+0000", 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636544+0000", 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636982+0000", 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043829999999999997 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637045+0000", 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3392999999999994e-05 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637057+0000", 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1559000000000001e-05 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637114+0000", 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-15T02:49:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637126+0000", 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1888e-05 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641808+0000", 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046821679999999996 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641841+0000", 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2787000000000003e-05 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.8689900010000002, 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085723550000000003, 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749730+0000", 2024-09-15T02:49:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1500000000000004e-07 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749937+0000", 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020695800000000001 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750030+0000", 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3487999999999997e-05 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750043+0000", 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2322e-05 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750162+0000", 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119388 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750182+0000", 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.02e-05 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758264+0000", 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080818000000000001 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758302+0000", 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7583999999999998e-05 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=180, reqid=client.4526.0:385, at_version=49'53, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.8591184050000003, 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010517344, 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759603+0000", 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7179999999999999e-06 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759826+0000", 2024-09-15T02:49:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0002231 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759971+0000", 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014553100000000001 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760158+0000", 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018628199999999999 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760305+0000", 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014718500000000001 2024-09-15T02:49:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760353+0000", 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8418999999999998e-05 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770084+0000", 2024-09-15T02:49:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097306449999999996 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770118+0000", 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4464000000000001e-05 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.857839061, 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078090939999999999, 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760882+0000", 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.145e-06 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761368+0000", 2024-09-15T02:49:00.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048694100000000001 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761447+0000", 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8586999999999998e-05 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761462+0000", 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4588e-05 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761530+0000", 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8718000000000002e-05 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761543+0000", 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.274e-05 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768640+0000", 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070968000000000003 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768689+0000", 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9574999999999999e-05 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 49/33 ECSubWrite(tid=48, reqid=client.4526.0:404, at_version=49'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.770958+0000", 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.8477611429999996, 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057426880000000001, 2024-09-15T02:49:00.680 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770958+0000", 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770958+0000", 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770959+0000", 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3600000000000002e-07 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771401+0000", 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044212100000000002 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771443+0000", 2024-09-15T02:49:00.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1940999999999997e-05 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771465+0000", 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771520+0000", 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4901000000000002e-05 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771535+0000", 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4552000000000001e-05 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776642+0000", 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005107388 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776701+0000", 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8829999999999997e-05 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.684 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=188, reqid=client.4526.0:389, at_version=49'57, trim_to=0'0, roll_forward_to=49'55))", 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.955686+0000", 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.6630330210000004, 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061702720000000001, 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955686+0000", 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955686+0000", 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955689+0000", 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3039999999999999e-06 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955797+0000", 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010848099999999999 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955911+0000", 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011391 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955933+0000", 2024-09-15T02:49:00.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2039999999999999e-05 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.956065+0000", 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131927 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.956095+0000", 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9589000000000001e-05 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.961823+0000", 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057285050000000001 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.961857+0000", 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3516000000000002e-05 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.689 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3461253710000003, 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090363309999999999, 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.34e-07 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272948+0000", 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000353715 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273002+0000", 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3458999999999998e-05 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273014+0000", 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1997000000000001e-05 2024-09-15T02:49:00.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273101+0000", 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7424999999999997e-05 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273113+0000", 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1558e-05 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.281599+0000", 2024-09-15T02:49:00.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084866360000000005 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.281630+0000", 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1207000000000001e-05 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:00.694 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3342794080000004, 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015418308, 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284441+0000", 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.251e-06 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284558+0000", 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011635 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284645+0000", 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6879000000000006e-05 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284653+0000", 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8100000000000004e-06 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284760+0000", 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106585 2024-09-15T02:49:00.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284774+0000", 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4260000000000001e-05 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299827+0000", 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015052345 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299858+0000", 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1828000000000001e-05 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=62, reqid=client.4526.0:418, at_version=50'6, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3341568060000002, 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015317982000000001, 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.699 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284595+0000", 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2381e-05 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284673+0000", 2024-09-15T02:49:00.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7855000000000006e-05 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284681+0000", 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9910000000000002e-06 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289497+0000", 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048157060000000003 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289509+0000", 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.166e-05 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299868+0000", 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010359454000000001 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299881+0000", 2024-09-15T02:49:00.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2622e-05 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=63, reqid=client.4526.0:419, at_version=50'7, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3292073230000003, 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014955008000000001, 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.704 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289513+0000", 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.569999999999999e-07 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289540+0000", 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6984000000000001e-05 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289594+0000", 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4117999999999999e-05 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289608+0000", 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4219999999999999e-05 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299620+0000", 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010011978 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299633+0000", 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2564e-05 2024-09-15T02:49:00.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304436+0000", 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048034269999999999 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304467+0000", 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0759999999999997e-05 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.708 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3277564269999997, 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014581759999999999, 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:00.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290964+0000", 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6499999999999998e-07 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291188+0000", 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224513 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291233+0000", 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4665e-05 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291241+0000", 2024-09-15T02:49:00.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8019999999999992e-06 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291323+0000", 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2284e-05 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291336+0000", 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2906e-05 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305516+0000", 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014179916000000001 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305545+0000", 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8809000000000001e-05 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.713 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3273683270000003, 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014221952, 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:00.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291352+0000", 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6000000000000004e-07 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291563+0000", 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000211387 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291616+0000", 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3294000000000002e-05 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291625+0000", 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8559999999999996e-06 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299094+0000", 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007469084 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299105+0000", 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0226e-05 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305552+0000", 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064471270000000004 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305573+0000", 2024-09-15T02:49:00.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1418000000000001e-05 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=202, reqid=client.4526.0:432, at_version=50'61, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.297119+0000", 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.321600374, 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084870129999999998, 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297119+0000", 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297119+0000", 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297120+0000", 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.381e-06 2024-09-15T02:49:00.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297159+0000", 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8161999999999998e-05 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297207+0000", 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8541000000000002e-05 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297215+0000", 2024-09-15T02:49:00.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7759999999999997e-06 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303660+0000", 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006445465 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303670+0000", 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8400000000000007e-06 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305587+0000", 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019168449999999999 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305606+0000", 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9003000000000001e-05 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.722 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=203, reqid=client.4526.0:433, at_version=50'62, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.298863+0000", 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3198564880000001, 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067607240000000001, 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298863+0000", 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298863+0000", 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298864+0000", 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1459999999999999e-06 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298891+0000", 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7118e-05 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298948+0000", 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7224e-05 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298959+0000", 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0706e-05 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304638+0000", 2024-09-15T02:49:00.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056788819999999997 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304648+0000", 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7769999999999997e-06 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305608+0000", 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096046499999999997 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305624+0000", 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5406000000000001e-05 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=204, reqid=client.4526.0:438, at_version=50'64, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.300097+0000", 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3186222819999998, 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063381089999999998, 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:00.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300097+0000", 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300097+0000", 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300098+0000", 2024-09-15T02:49:00.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4200000000000005e-07 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300121+0000", 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2884e-05 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300187+0000", 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6013000000000006e-05 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300194+0000", 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0700000000000001e-06 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304854+0000", 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004659959 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304864+0000", 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6430000000000007e-06 2024-09-15T02:49:00.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306408+0000", 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001544562 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306435+0000", 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7135999999999999e-05 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:00.731 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:00.734 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:00.729+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:00.834 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:00.829+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:00.841 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:00.841 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.851 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:00.935 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:00.930+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:00.953 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:00.953 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:00.953 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:00.953 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:00.953 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:00.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:01.037 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:01.033+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:01.065 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:01.065 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:01.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4311.0:7 2.2 e29/26 2:5cea7035:::rbd_info:head v 29'2, mlcod=0'0)", 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.641200291000001, 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017517679999999999, 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:49:01.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422418+0000", 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422427+0000", 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.473e-06 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422543+0000", 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115732 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422561+0000", 2024-09-15T02:49:01.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8505e-05 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422669+0000", 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108175 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.422706+0000", 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6851999999999999e-05 2024-09-15T02:49:01.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.424081+0000", 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013743970000000001 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:29.424170+0000", 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9098000000000007e-05 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 39/33 ECSubWrite(tid=16, reqid=client.4526.0:231, at_version=39'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.489835338999999, 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00223799, 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:49:01.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573783+0000", 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9699999999999996e-07 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573829+0000", 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5249e-05 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573916+0000", 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6917999999999997e-05 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.573931+0000", 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5546e-05 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574047+0000", 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011535300000000001 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574060+0000", 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3016999999999999e-05 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575974+0000", 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019143750000000001 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576021+0000", 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6934999999999997e-05 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 39/33 ECSubWrite(tid=17, reqid=client.4526.0:232, at_version=39'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:43.574020+0000", 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.489597987, 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025056259999999999, 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.076 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574020+0000", 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574020+0000", 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574021+0000", 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1200000000000002e-07 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574044+0000", 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3465999999999999e-05 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574087+0000", 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2979999999999998e-05 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.574096+0000", 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5969999999999992e-06 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575374+0000", 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00127786 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.575390+0000", 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5619000000000001e-05 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576491+0000", 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011011459999999999 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:43.576526+0000", 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5145999999999999e-05 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=20, reqid=client.4526.0:249, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:01.080 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.471211500000001, 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020019123999999999, 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592951+0000", 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054339899999999997 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593028+0000", 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6892999999999997e-05 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593044+0000", 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6407999999999999e-05 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595661+0000", 2024-09-15T02:49:01.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002617032 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595686+0000", 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5358999999999999e-05 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612387+0000", 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016700194000000002 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612426+0000", 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9138999999999999e-05 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=21, reqid=client.4526.0:250, at_version=41'14, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.459298749, 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086779219999999994, 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.085 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604321+0000", 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.902e-06 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604517+0000", 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019603099999999999 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604572+0000", 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4144000000000002e-05 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604581+0000", 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4360000000000008e-06 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608062+0000", 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034810710000000001 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608079+0000", 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7059999999999999e-05 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612964+0000", 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004885197 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612997+0000", 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3080999999999997e-05 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=22, reqid=client.4526.0:251, at_version=41'15, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.608374+0000", 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.455244349999999, 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049527460000000001, 2024-09-15T02:49:01.090 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608374+0000", 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608374+0000", 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608375+0000", 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608397+0000", 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1888000000000001e-05 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608443+0000", 2024-09-15T02:49:01.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6020000000000003e-05 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608451+0000", 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.456e-06 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612489+0000", 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004037629 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612499+0000", 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0363000000000001e-05 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.613275+0000", 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077597099999999997 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.613327+0000", 2024-09-15T02:49:01.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1484999999999999e-05 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/33 ECSubWrite(tid=36, reqid=client.4526.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=43'21))", 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.372910+0000", 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.690707742000001, 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018596559999999999, 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372910+0000", 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372910+0000", 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372912+0000", 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8470000000000001e-06 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.372948+0000", 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5621999999999999e-05 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373050+0000", 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101875 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373069+0000", 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8706e-05 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373225+0000", 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015610600000000001 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.373260+0000", 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4965999999999997e-05 2024-09-15T02:49:01.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.374697+0000", 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001437345 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.374770+0000", 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3188999999999993e-05 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=29, reqid=osd.8.0:29, at_version=44'21, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.465478461, 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004464193, 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598140+0000", 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598197+0000", 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6575e-05 2024-09-15T02:49:01.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598305+0000", 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010784400000000001 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598319+0000", 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3987e-05 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598448+0000", 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000129481 2024-09-15T02:49:01.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.598479+0000", 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0629000000000001e-05 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602507+0000", 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040278609999999998 2024-09-15T02:49:01.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.602604+0000", 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7145999999999995e-05 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.463652074000001, 2024-09-15T02:49:01.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080113409999999999, 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599967+0000", 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3120000000000001e-06 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600013+0000", 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126999999999997e-05 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600081+0000", 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8295999999999998e-05 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600094+0000", 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3522e-05 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600358+0000", 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026319700000000001 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600388+0000", 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029e-05 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607948+0000", 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075605949999999998 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607977+0000", 2024-09-15T02:49:01.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9263000000000001e-05 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=31, reqid=osd.8.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.609028+0000", 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.454589845999999, 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023506960000000002, 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609028+0000", 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609028+0000", 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609030+0000", 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1969999999999999e-06 2024-09-15T02:49:01.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609043+0000", 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3813e-05 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609098+0000", 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5025999999999998e-05 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609108+0000", 2024-09-15T02:49:01.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2809999999999997e-06 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609631+0000", 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052322800000000004 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.609641+0000", 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0206e-05 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.611349+0000", 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017074239999999999 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.611379+0000", 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0521000000000001e-05 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.113 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/33 ECSubWrite(tid=39, reqid=osd.14.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=44'25))", 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.346169+0000", 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.71744955, 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022908160000000002, 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346169+0000", 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346169+0000", 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346171+0000", 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069999999999998e-06 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346195+0000", 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4604999999999999e-05 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346288+0000", 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2881999999999996e-05 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346304+0000", 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5373e-05 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346473+0000", 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000169603 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.346507+0000", 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3442000000000003e-05 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.348384+0000", 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018776820000000001 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.348459+0000", 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5022000000000001e-05 2024-09-15T02:49:01.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.426944366000001, 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050988680000000003, 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.636674+0000", 2024-09-15T02:49:01.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2399999999999999e-07 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637113+0000", 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043827200000000002 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637157+0000", 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4824999999999999e-05 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637167+0000", 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7950000000000002e-06 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637247+0000", 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0315000000000006e-05 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.637263+0000", 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5156e-05 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641738+0000", 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044751230000000001 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.641773+0000", 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4958e-05 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.122 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 50/33 ECSubWrite(tid=36, reqid=client.4526.0:421, at_version=50'7, trim_to=0'0, roll_forward_to=36'5))", 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.269071+0000", 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.7945470419999996, 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022646060000000002, 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269071+0000", 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269071+0000", 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269072+0000", 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4799999999999999e-07 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269131+0000", 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9839000000000001e-05 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3114999999999994e-05 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269217+0000", 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3029e-05 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269306+0000", 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8188000000000003e-05 2024-09-15T02:49:01.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269325+0000", 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9264000000000001e-05 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271283+0000", 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019582879999999999 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271336+0000", 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2534999999999998e-05 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.792727438, 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012340179999999999, 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0999999999999999e-07 2024-09-15T02:49:01.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271155+0000", 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026426800000000001 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271179+0000", 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3893999999999999e-05 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271190+0000", 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0525e-05 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271256+0000", 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6328999999999995e-05 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271268+0000", 2024-09-15T02:49:01.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1831000000000001e-05 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283197+0000", 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011929294999999999 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283231+0000", 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3627999999999997e-05 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.131 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.278363+0000", 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.78525562, 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051390910000000001, 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278363+0000", 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278363+0000", 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278364+0000", 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.139e-06 2024-09-15T02:49:01.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278542+0000", 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017818700000000001 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278581+0000", 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9258999999999998e-05 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278588+0000", 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3220000000000004e-06 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278780+0000", 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191681 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278787+0000", 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3990000000000001e-06 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283463+0000", 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046766899999999998 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283502+0000", 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8414000000000002e-05 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=62, reqid=client.4526.0:418, at_version=50'6, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284004+0000", 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.7796137439999997, 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032969980000000002, 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284004+0000", 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284004+0000", 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284006+0000", 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284120+0000", 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114286 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284187+0000", 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7119000000000006e-05 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284195+0000", 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0639999999999994e-06 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284257+0000", 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1577999999999999e-05 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284270+0000", 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3038e-05 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287270+0000", 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003000674 2024-09-15T02:49:01.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287301+0000", 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0994000000000002e-05 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=66, reqid=client.4526.0:443, at_version=51'9, trim_to=0'0, roll_forward_to=50'7))", 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:55.236535+0000", 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8270828950000002, 2024-09-15T02:49:01.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021857069999999998, 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236535+0000", 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236535+0000", 2024-09-15T02:49:01.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236536+0000", 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236602+0000", 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6438000000000003e-05 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236726+0000", 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012377800000000001 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236748+0000", 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2351999999999999e-05 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236862+0000", 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113649 2024-09-15T02:49:01.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.236882+0000", 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0333e-05 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.238676+0000", 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017938369999999999 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.238721+0000", 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4693000000000001e-05 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 51/33 ECSubWrite(tid=67, reqid=client.4526.0:444, at_version=51'10, trim_to=0'0, roll_forward_to=50'7))", 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:55.239340+0000", 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8242777769999998, 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002756175, 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.146 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239340+0000", 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239340+0000", 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239342+0000", 2024-09-15T02:49:01.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3260000000000001e-06 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239381+0000", 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9076999999999999e-05 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239453+0000", 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1692000000000004e-05 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239465+0000", 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2629999999999999e-05 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239556+0000", 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1293999999999997e-05 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.239580+0000", 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3312999999999999e-05 2024-09-15T02:49:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.242015+0000", 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024349860000000001 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.242097+0000", 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1856999999999999e-05 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 52/33 ECSubWrite(tid=70, reqid=client.4526.0:455, at_version=52'34, trim_to=0'0, roll_forward_to=48'32))", 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:56.253868+0000", 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.8097497469999997, 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031341350000000001, 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.253868+0000", 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.253868+0000", 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.253869+0000", 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8599999999999999e-07 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.253930+0000", 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0871e-05 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.254029+0000", 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9438000000000006e-05 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.254044+0000", 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5169e-05 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.254172+0000", 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012799600000000001 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.254197+0000", 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4457e-05 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.256974+0000", 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027776089999999999 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.257003+0000", 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8309e-05 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 52/33 ECSubWrite(tid=71, reqid=client.4526.0:456, at_version=52'35, trim_to=0'0, roll_forward_to=48'32))", 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:56.255463+0000", 2024-09-15T02:49:01.155 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.808155653, 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021416149999999999, 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.255463+0000", 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.255463+0000", 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.255464+0000", 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.252e-06 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.255506+0000", 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1718999999999998e-05 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.255578+0000", 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2188999999999996e-05 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.255588+0000", 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0723999999999999e-05 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.256006+0000", 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000418033 2024-09-15T02:49:01.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.256020+0000", 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3664e-05 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.257576+0000", 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001555648 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:56.257604+0000", 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8385999999999999e-05 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.160 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:01.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:01.163 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:01.158+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:01.263 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:01.257+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:01.268 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:01.268 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:01.268 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:01.268 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:01.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:01.295 INFO:tasks.rados.rados.0.smithi191.stdout:185: done (0 left) 2024-09-15T02:49:01.295 INFO:tasks.rados.rados.0.smithi191.stdout:186: rollback oid 43 current snap is 14 2024-09-15T02:49:01.295 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 43 to 7 2024-09-15T02:49:01.295 INFO:tasks.rados.rados.0.smithi191.stdout:187: read oid 14 snap -1 2024-09-15T02:49:01.295 INFO:tasks.rados.rados.0.smithi191.stdout:187: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-15T02:49:01.296 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 28 current snap is 14 2024-09-15T02:49:01.296 INFO:tasks.rados.rados.0.smithi191.stdout:188: seq_num 90 ranges {442368=753664,1196032=720896,1916928=49152} 2024-09-15T02:49:01.308 INFO:tasks.rados.rados.0.smithi191.stdout:188: writing smithi19125844-28 from 442368 to 1196032 tid 1 2024-09-15T02:49:01.313 INFO:tasks.rados.rados.0.smithi191.stdout:188: writing smithi19125844-28 from 1196032 to 1916928 tid 2 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:188: writing smithi19125844-28 from 1916928 to 1966080 tid 3 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:189: rollback oid 32 current snap is 14 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 32 to 7 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:186: finishing rollback tid 0 to smithi19125844-43 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:186: finishing rollback tid 1 to smithi19125844-43 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 29 (ObjNum 145 snap 0 seq_num 565500240) dirty dne 2024-09-15T02:49:01.314 INFO:tasks.rados.rados.0.smithi191.stdout:187: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-15T02:49:01.340 INFO:tasks.rados.rados.0.smithi191.stdout:189: finishing rollback tid 0 to smithi19125844-32 2024-09-15T02:49:01.341 INFO:tasks.rados.rados.0.smithi191.stdout:189: finishing rollback tid 1 to smithi19125844-32 2024-09-15T02:49:01.341 INFO:tasks.rados.rados.0.smithi191.stdout:189: finishing rollback tid 2 to smithi19125844-32 2024-09-15T02:49:01.341 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 32 v 33 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T02:49:01.341 INFO:tasks.rados.rados.0.smithi191.stdout:188: finishing write tid 1 to smithi19125844-28 2024-09-15T02:49:01.341 INFO:tasks.rados.rados.0.smithi191.stdout:186: done (3 left) 2024-09-15T02:49:01.341 INFO:tasks.rados.rados.0.smithi191.stdout:187: done (2 left) 2024-09-15T02:49:01.342 INFO:tasks.rados.rados.0.smithi191.stdout:189: done (1 left) 2024-09-15T02:49:01.342 INFO:tasks.rados.rados.0.smithi191.stdout:190: read oid 37 snap -1 2024-09-15T02:49:01.342 INFO:tasks.rados.rados.0.smithi191.stdout:190: expect (ObjNum 72 snap 7 seq_num 72) 2024-09-15T02:49:01.342 INFO:tasks.rados.rados.0.smithi191.stdout:191: delete oid 24 current snap is 14 2024-09-15T02:49:01.343 INFO:tasks.rados.rados.0.smithi191.stdout:188: finishing write tid 2 to smithi19125844-28 2024-09-15T02:49:01.344 INFO:tasks.rados.rados.0.smithi191.stdout:188: finishing write tid 3 to smithi19125844-28 2024-09-15T02:49:01.344 INFO:tasks.rados.rados.0.smithi191.stdout:188: finishing write tid 4 to smithi19125844-28 2024-09-15T02:49:01.345 INFO:tasks.rados.rados.0.smithi191.stdout:191: done (2 left) 2024-09-15T02:49:01.345 INFO:tasks.rados.rados.0.smithi191.stdout:192: read oid 39 snap 5 2024-09-15T02:49:01.346 INFO:tasks.rados.rados.0.smithi191.stdout:192: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-15T02:49:01.346 INFO:tasks.rados.rados.0.smithi191.stdout:193: rollback oid 10 current snap is 14 2024-09-15T02:49:01.346 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 10 to 9 2024-09-15T02:49:01.346 INFO:tasks.rados.rados.0.smithi191.stdout:194: setattr oid 12 current snap is 14 2024-09-15T02:49:01.348 INFO:tasks.rados.rados.0.smithi191.stdout:195: snap_remove snap 11 2024-09-15T02:49:01.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:49:01.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:01.364 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:01.362+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:01.424 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:01.425 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:01.433 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:01.465 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:01.462+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:01.491 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:01.520 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:388 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:01.531 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.7748937080000005, 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.205927366, 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 388 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:01.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752805+0000", 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8480000000000003e-06 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752820+0000", 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752827+0000", 2024-09-15T02:49:01.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8059999999999997e-06 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770449+0000", 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017621527000000001 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770567+0000", 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118576 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776872+0000", 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063044340000000003 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776986+0000", 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011429699999999999 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958728+0000", 2024-09-15T02:49:01.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.181741654 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:389 3.7s0 3:e10d9d4d:::smithi19125844-8:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.7748584469999997, 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47945425899999999, 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.537 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 389 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752836+0000", 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752842+0000", 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8579999999999996e-06 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752860+0000", 2024-09-15T02:49:01.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7816e-05 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752867+0000", 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0260000000000002e-06 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770793+0000", 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017925963999999999 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770931+0000", 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137978 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955584+0000", 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.184652765 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.959571+0000", 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003987488 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232070+0000", 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.272499243 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232290+0000", 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021937500000000001 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:390 3.7s0 3:e10d9d4d:::smithi19125844-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.774820257, 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20558775300000001, 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 390 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:01.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8199999999999999e-07 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752878+0000", 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4489999999999999e-06 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752888+0000", 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0122000000000001e-05 2024-09-15T02:49:01.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752894+0000", 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9220000000000002e-06 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771143+0000", 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018248665000000001 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771198+0000", 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5158000000000003e-05 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958258+0000", 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18706018699999999 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958359+0000", 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101025 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958461+0000", 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102543 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:402 3.1s0 3:801a1da7:::smithi19125844-39:e [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.7694424939999998, 2024-09-15T02:49:01.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52960550699999998, 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 402 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:01.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758252+0000", 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5000000000000002e-07 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758257+0000", 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3800000000000004e-06 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758267+0000", 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0455000000000001e-05 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758604+0000", 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033705300000000003 2024-09-15T02:49:01.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761065+0000", 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024613840000000001 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761208+0000", 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014282499999999999 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287857+0000", 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52664865999999999 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 49/33 ECSubReadReply(tid=183, attrs_read=0))", 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.7644699939999997, 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52471420199999996, 2024-09-15T02:49:01.553 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763572+0000", 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034791800000000001 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286378+0000", 2024-09-15T02:49:01.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52280558799999999 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286384+0000", 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286558+0000", 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017370499999999999 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287938+0000", 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013805550000000001 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:409 3.1s0 3:8610d5f9:::smithi19125844-1:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.251674+0000", 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2760199390000002, 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040551622000000002, 2024-09-15T02:49:01.557 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 409 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.251674+0000", 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.251674+0000", 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.251675+0000", 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5030000000000001e-06 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252402+0000", 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00072673699999999998 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252427+0000", 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4392999999999998e-05 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252439+0000", 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2072000000000001e-05 2024-09-15T02:49:01.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252564+0000", 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012556500000000001 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.252798+0000", 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023401699999999999 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.254690+0000", 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018922279999999999 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.269207+0000", 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014516625999999999 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292131+0000", 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022924243 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.292226+0000", 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4238000000000001e-05 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:412 3.1s0 3:8610d5f9:::smithi19125844-1:head [setxattr _header (55) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.256557+0000", 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2711372220000001, 2024-09-15T02:49:01.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.061249245000000001, 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 412 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256557+0000", 2024-09-15T02:49:01.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256557+0000", 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256558+0000", 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7400000000000002e-07 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256563+0000", 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1170000000000003e-06 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256572+0000", 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7210000000000006e-06 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256579+0000", 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3149999999999997e-06 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276049+0000", 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019470022999999999 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276235+0000", 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018618899999999999 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276938+0000", 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000703333 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277968+0000", 2024-09-15T02:49:01.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001029807 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.317677+0000", 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039708808999999998 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.317806+0000", 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012905699999999999 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:413 3.1s0 3:8610d5f9:::smithi19125844-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2711083570000001, 2024-09-15T02:49:01.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063143321000000002, 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 413 2024-09-15T02:49:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:49:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256586+0000", 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256595+0000", 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.473e-06 2024-09-15T02:49:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256608+0000", 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3523000000000001e-05 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.256615+0000", 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8480000000000003e-06 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278005+0000", 2024-09-15T02:49:01.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021389925000000001 2024-09-15T02:49:01.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:01.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.278191+0000", 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018626800000000001 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.318427+0000", 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040235663999999997 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.318542+0000", 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011523000000000001 2024-09-15T02:49:01.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319729+0000", 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011866540000000001 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:433 3.7s0 3:ec23d144:::smithi19125844-41:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:01.578 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2447006209999998, 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27191602199999998, 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-15T02:49:01.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282994+0000", 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1790000000000001e-06 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283000+0000", 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0479999999999998e-06 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283008+0000", 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1440000000000003e-06 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283012+0000", 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7129999999999996e-06 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298201+0000", 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015188495 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298355+0000", 2024-09-15T02:49:01.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015441200000000001 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298951+0000", 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059618300000000002 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300677+0000", 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001725285 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554804+0000", 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25412710900000002 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554909+0000", 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105454 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:434 3.7s0 3:ec23d144:::smithi19125844-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2446766980000001, 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27651220700000001, 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.586 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 434 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283018+0000", 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0500000000000003e-07 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283021+0000", 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.959e-06 2024-09-15T02:49:01.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283028+0000", 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0169999999999999e-06 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283032+0000", 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0509999999999998e-06 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299373+0000", 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016340659 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299481+0000", 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108218 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554991+0000", 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25551037700000001 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.555256+0000", 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026495400000000002 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559529+0000", 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042733670000000001 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.591 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:437 3.6s0 3:6e9f7d10:::smithi19125844-33:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2446097610000004, 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20344801100000001, 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 437 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283085+0000", 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283088+0000", 2024-09-15T02:49:01.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.993e-06 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283094+0000", 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5810000000000002e-06 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6289999999999999e-06 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284035+0000", 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093763400000000002 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284100+0000", 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4882999999999995e-05 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486532+0000", 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20243185599999999 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:438 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [delete] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e50)", 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2445759519999999, 2024-09-15T02:49:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27354936000000002, 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 438 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8100000000000003e-07 2024-09-15T02:49:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283121+0000", 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8830000000000002e-06 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283129+0000", 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3390000000000004e-06 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283133+0000", 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299523+0000", 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389989000000001 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299625+0000", 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010208 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300116+0000", 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049149000000000003 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300972+0000", 2024-09-15T02:49:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085554200000000004 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556582+0000", 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25561025399999998 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556667+0000", 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5208000000000001e-05 2024-09-15T02:49:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.6s0 50/33 ECSubReadReply(tid=200, attrs_read=0))", 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2324063589999996, 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19127902199999999, 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2600000000000001e-07 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295338+0000", 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9951999999999999e-05 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485520+0000", 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19018270600000001 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485530+0000", 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0820000000000005e-06 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485655+0000", 2024-09-15T02:49:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012580999999999999 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486567+0000", 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000911246 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=203, last_complete=50'62, committed=1, applied=1))", 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2243611769999996, 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25159346799999999, 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303336+0000", 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9550000000000001e-06 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554637+0000", 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25130088299999997 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554647+0000", 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9329999999999996e-06 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554704+0000", 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6514000000000001e-05 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554926+0000", 2024-09-15T02:49:01.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222647 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 50/33 ECSubWriteReply(tid=205, last_complete=50'35, committed=1, applied=1))", 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.2083537470000003, 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.23186345, 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.613 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319341+0000", 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6700000000000003e-07 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319347+0000", 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6879999999999996e-06 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550925+0000", 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.231578591 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550946+0000", 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0903e-05 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551110+0000", 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016408300000000001 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551204+0000", 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3418000000000006e-05 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=208, last_complete=50'64, committed=1, applied=1))", 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.9684965810000001, 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.60425413100000003, 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559198+0000", 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9500000000000001e-07 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559203+0000", 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.13e-06 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163222+0000", 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.604018638 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163237+0000", 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4622e-05 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163384+0000", 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014714099999999999 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163451+0000", 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7805000000000006e-05 2024-09-15T02:49:01.621 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 54/53 ECSubWrite(tid=87, reqid=client.4526.0:461, at_version=0'0, trim_to=0'0, roll_forward_to=54'39))", 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.6065939139999998, 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.41018363000000002, 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921102+0000", 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921127+0000", 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4987e-05 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329820+0000", 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.40869367400000001 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329833+0000", 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2669e-05 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329958+0000", 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012490099999999999 2024-09-15T02:49:01.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329981+0000", 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2463000000000002e-05 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331170+0000", 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001189701 2024-09-15T02:49:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331284+0000", 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113435 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:467 3.bs0 3:d4a23ffb:::smithi19125844-23:head [stat,delete] snapc 13=[13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:01.627 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.274765+0000", 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.2529287280000001, 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040340187, 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 467 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274765+0000", 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274765+0000", 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274768+0000", 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3099999999999999e-06 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274779+0000", 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1443e-05 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274809+0000", 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0389e-05 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274821+0000", 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.199e-05 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274947+0000", 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012531400000000001 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.274959+0000", 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2356e-05 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.279284+0000", 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043251519999999996 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.279418+0000", 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013413300000000001 2024-09-15T02:49:01.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.315105+0000", 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035687099999999999 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPushReply(3.1s0 54/53 [PushReplyOp(3:801a1da7:::smithi19125844-39:head)])", 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.232877131, 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99244340499999995, 2024-09-15T02:49:01.633 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294818+0000", 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294837+0000", 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9052999999999999e-05 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286615+0000", 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99177708899999995 2024-09-15T02:49:01.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286629+0000", 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4583000000000001e-05 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286721+0000", 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1640999999999999e-05 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.287260+0000", 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053939199999999997 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 55/53 ECSubRead(tid=88, to_read={3:94f52c52:::smithi19125844-20:head=0,110592,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.1978066580000002, 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59246917700000001, 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:01.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329888+0000", 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-09-15T02:49:01.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329900+0000", 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.188e-05 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921816+0000", 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59191607099999999 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921831+0000", 2024-09-15T02:49:01.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4765e-05 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922011+0000", 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018023900000000001 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922356+0000", 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034564200000000001 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:01.641 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:01.641 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:01.642 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:01.642 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:01.642 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:01.642 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:01.643 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:01.643 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:01.641+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:01.664 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:01.693 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:01.722 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:01.743 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:01.735+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:01.749 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:01.751 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:01.751 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:01.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:01.782 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:01.811 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:01.842 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:01.843 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:01.838+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:01.857 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:01.857 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:01.866 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:01.870 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:01.871+0000 7f8e7a7fc700 0 -- 172.21.15.6:0/341209402 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f8e84193090 msgr2=0x7f8e84195470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:01.908 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:01.939 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:01.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:01.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:01.944 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:01.939+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:01.964 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:01.965 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:01.973 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:02.045 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:02.045+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:02.073 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.073 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.083 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:49:02.146 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:02.141+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:02.184 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.184 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.193 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:49:02.247 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:02.243+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:02.274 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 10] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:49:02.274 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:49:02.274 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-15T02:49:02.274 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.527322 2024-09-15T02:49:02.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0.5273222173835144 2024-09-15T02:49:02.288 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.288 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.297 INFO:tasks.rados.rados.0.smithi191.stdout:195: done (5 left) 2024-09-15T02:49:02.297 INFO:tasks.rados.rados.0.smithi191.stdout:196: read oid 42 snap -1 2024-09-15T02:49:02.297 INFO:tasks.rados.rados.0.smithi191.stdout:196: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T02:49:02.298 INFO:tasks.rados.rados.0.smithi191.stdout:197: snap_remove snap 7 2024-09-15T02:49:02.299 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:02.302 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:49:02.348 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:02.346+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:02.390 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.390 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:02.450 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:02.445+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:02.498 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.498 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.507 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:02.550 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:02.545+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:02.603 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.603 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:02.651 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:02.645+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:02.717 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.717 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:02.752 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:02.749+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:02.824 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.824 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:02.853 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:02.850+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:02.935 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:02.935 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:02.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:02.953 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:02.950+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:03.042 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.042 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:03.054 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:03.049+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:03.154 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.154 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.155 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:03.149+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:03.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:03.256 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:03.254+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:03.263 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.263 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:03.313 INFO:tasks.rados.rados.0.smithi191.stdout:197: done (6 left) 2024-09-15T02:49:03.313 INFO:tasks.rados.rados.0.smithi191.stdout:198: snap_create 2024-09-15T02:49:03.313 INFO:tasks.rados.rados.0.smithi191.stdout:188: finishing write tid 5 to smithi19125844-28 2024-09-15T02:49:03.313 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 26 (ObjNum 90 snap 14 seq_num 90) dirty exists 2024-09-15T02:49:03.313 INFO:tasks.rados.rados.0.smithi191.stdout:188: left oid 28 (ObjNum 90 snap 14 seq_num 90) 2024-09-15T02:49:03.313 INFO:tasks.rados.rados.0.smithi191.stdout:192: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-15T02:49:03.318 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.6 primary-affinity to 0.527322 (8345582) 2024-09-15T02:49:03.344 INFO:tasks.rados.rados.0.smithi191.stdout:193: finishing rollback tid 0 to smithi19125844-10 2024-09-15T02:49:03.344 INFO:tasks.rados.rados.0.smithi191.stdout:193: finishing rollback tid 1 to smithi19125844-10 2024-09-15T02:49:03.344 INFO:tasks.rados.rados.0.smithi191.stdout:193: finishing rollback tid 2 to smithi19125844-10 2024-09-15T02:49:03.344 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 10 v 38 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T02:49:03.345 INFO:tasks.rados.rados.0.smithi191.stdout:190: expect (ObjNum 72 snap 7 seq_num 72) 2024-09-15T02:49:03.357 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:03.353+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:03.379 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.379 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:03.423 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 12 v 40 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T02:49:03.423 INFO:tasks.rados.rados.0.smithi191.stdout:196: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T02:49:03.458 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:03.453+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:03.485 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.485 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:03.559 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:03.554+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:03.593 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.593 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:03.660 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:03.658+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:03.697 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.697 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:03.761 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:03.758+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:03.804 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.804 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:03.861 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:03.857+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:03.913 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:03.913 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:03.922 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:03.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:03.962 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:03.957+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:04.031 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:04.032 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:04.042 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:04.063 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:04.055+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:04.121 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:04.138 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:04.138 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:04.147 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:04.151 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:04.164 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:04.155+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:04.179 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:04.208 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:04.232 INFO:tasks.rados.rados.0.smithi191.stdout:188: done (6 left) 2024-09-15T02:49:04.232 INFO:tasks.rados.rados.0.smithi191.stdout:190: done (5 left) 2024-09-15T02:49:04.232 INFO:tasks.rados.rados.0.smithi191.stdout:192: done (4 left) 2024-09-15T02:49:04.232 INFO:tasks.rados.rados.0.smithi191.stdout:193: done (3 left) 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:194: done (2 left) 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:196: done (1 left) 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:198: done (0 left) 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:199: rollback oid 44 current snap is 15 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 44 to 4 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:200: read oid 11 snap -1 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:200: expect (ObjNum 83 snap 9 seq_num 83) 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 7 current snap is 15 2024-09-15T02:49:04.233 INFO:tasks.rados.rados.0.smithi191.stdout:201: seq_num 92 ranges {114688=786432,901120=491520,1392640=278528} 2024-09-15T02:49:04.237 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:04.244 INFO:tasks.rados.rados.0.smithi191.stdout:201: writing smithi19125844-7 from 114688 to 901120 tid 1 2024-09-15T02:49:04.246 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:04.246 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:04.247 INFO:tasks.rados.rados.0.smithi191.stdout:201: writing smithi19125844-7 from 901120 to 1392640 tid 2 2024-09-15T02:49:04.249 INFO:tasks.rados.rados.0.smithi191.stdout:201: writing smithi19125844-7 from 1392640 to 1671168 tid 3 2024-09-15T02:49:04.249 INFO:tasks.rados.rados.0.smithi191.stdout:202: read oid 5 snap -1 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:202: expect (ObjNum 68 snap 5 seq_num 68) 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:203: rollback oid 6 current snap is 15 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 6 to 13 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:199: finishing rollback tid 0 to smithi19125844-44 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:204: rollback oid 32 current snap is 15 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 32 to 13 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:205: read oid 16 snap -1 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:205: expect deleted 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 10 current snap is 15 2024-09-15T02:49:04.250 INFO:tasks.rados.rados.0.smithi191.stdout:206: seq_num 93 ranges {163840=802816,966656=638976,1605632=131072} 2024-09-15T02:49:04.255 INFO:tasks.rados.rados.0.smithi191.stdout:206: writing smithi19125844-10 from 163840 to 966656 tid 1 2024-09-15T02:49:04.261 INFO:tasks.rados.rados.0.smithi191.stdout:206: writing smithi19125844-10 from 966656 to 1605632 tid 2 2024-09-15T02:49:04.261 INFO:tasks.rados.rados.0.smithi191.stdout:206: writing smithi19125844-10 from 1605632 to 1736704 tid 3 2024-09-15T02:49:04.261 INFO:tasks.rados.rados.0.smithi191.stdout:207: delete oid 15 current snap is 15 2024-09-15T02:49:04.261 INFO:tasks.rados.rados.0.smithi191.stdout:199: finishing rollback tid 1 to smithi19125844-44 2024-09-15T02:49:04.261 INFO:tasks.rados.rados.0.smithi191.stdout:199: finishing rollback tid 2 to smithi19125844-44 2024-09-15T02:49:04.262 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 44 v 27 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T02:49:04.262 INFO:tasks.rados.rados.0.smithi191.stdout:200: expect (ObjNum 83 snap 9 seq_num 83) 2024-09-15T02:49:04.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:04.265 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:04.257+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:04.269 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:04.295 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:04.310 INFO:tasks.rados.rados.0.smithi191.stdout:204: finishing rollback tid 0 to smithi19125844-32 2024-09-15T02:49:04.310 INFO:tasks.rados.rados.0.smithi191.stdout:204: finishing rollback tid 1 to smithi19125844-32 2024-09-15T02:49:04.310 INFO:tasks.rados.rados.0.smithi191.stdout:204: finishing rollback tid 2 to smithi19125844-32 2024-09-15T02:49:04.310 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 32 v 45 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T02:49:04.310 INFO:tasks.rados.rados.0.smithi191.stdout:202: expect (ObjNum 68 snap 5 seq_num 68) 2024-09-15T02:49:04.321 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:04.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:49:04.350 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:04.364 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:04.364 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:04.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.364 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:522 3.fs0 3:f39d968a:::smithi19125844-6:head [append 0~0] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.364 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.11332708700000001, 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.113377336, 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 522 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.365 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249055+0000", 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0800000000000005e-07 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249058+0000", 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.914e-06 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249068+0000", 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0883000000000001e-05 2024-09-15T02:49:04.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249073+0000", 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5399999999999997e-06 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.265365+0000", 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016292469 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.265485+0000", 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011935 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266071+0000", 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058648000000000001 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271231+0000", 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051601140000000004 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.369 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.370 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-15T02:49:04.370 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 1 2024-09-15T02:49:04.370 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:04.370 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:04.357+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:04.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:04.376 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:201: finishing write tid 1 to smithi19125844-7 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:201: finishing write tid 2 to smithi19125844-7 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:201: finishing write tid 3 to smithi19125844-7 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:201: finishing write tid 4 to smithi19125844-7 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:203: finishing rollback tid 0 to smithi19125844-6 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:203: finishing rollback tid 1 to smithi19125844-6 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:201: finishing write tid 5 to smithi19125844-7 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 7 v 27 (ObjNum 92 snap 15 seq_num 92) dirty exists 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:201: left oid 7 (ObjNum 92 snap 15 seq_num 92) 2024-09-15T02:49:04.381 INFO:tasks.rados.rados.0.smithi191.stdout:206: finishing write tid 1 to smithi19125844-10 2024-09-15T02:49:04.382 INFO:tasks.rados.rados.0.smithi191.stdout:206: finishing write tid 2 to smithi19125844-10 2024-09-15T02:49:04.382 INFO:tasks.rados.rados.0.smithi191.stdout:206: finishing write tid 3 to smithi19125844-10 2024-09-15T02:49:04.382 INFO:tasks.rados.rados.0.smithi191.stdout:206: finishing write tid 4 to smithi19125844-10 2024-09-15T02:49:04.382 INFO:tasks.rados.rados.0.smithi191.stdout:206: finishing write tid 5 to smithi19125844-10 2024-09-15T02:49:04.382 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 10 v 55 (ObjNum 93 snap 15 seq_num 93) dirty exists 2024-09-15T02:49:04.382 INFO:tasks.rados.rados.0.smithi191.stdout:206: left oid 10 (ObjNum 93 snap 15 seq_num 93) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:199: done (8 left) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:200: done (7 left) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:201: done (6 left) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:202: done (5 left) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:204: done (4 left) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:205: done (3 left) 2024-09-15T02:49:04.394 INFO:tasks.rados.rados.0.smithi191.stdout:206: done (2 left) 2024-09-15T02:49:04.395 INFO:tasks.rados.rados.0.smithi191.stdout:207: done (1 left) 2024-09-15T02:49:04.395 INFO:tasks.rados.rados.0.smithi191.stdout:208: delete oid 33 current snap is 15 2024-09-15T02:49:04.399 INFO:tasks.rados.rados.0.smithi191.stdout:208: done (1 left) 2024-09-15T02:49:04.399 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 42 current snap is 15 2024-09-15T02:49:04.399 INFO:tasks.rados.rados.0.smithi191.stdout:209: seq_num 96 ranges {704512=458752,1163264=688128,1851392=475136} 2024-09-15T02:49:04.406 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:04.411 INFO:tasks.rados.rados.0.smithi191.stdout:209: writing smithi19125844-42 from 704512 to 1163264 tid 1 2024-09-15T02:49:04.416 INFO:tasks.rados.rados.0.smithi191.stdout:209: writing smithi19125844-42 from 1163264 to 1851392 tid 2 2024-09-15T02:49:04.419 INFO:tasks.rados.rados.0.smithi191.stdout:209: writing smithi19125844-42 from 1851392 to 2326528 tid 3 2024-09-15T02:49:04.419 INFO:tasks.rados.rados.0.smithi191.stdout:210: rollback oid 3 current snap is 15 2024-09-15T02:49:04.419 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 3 to 14 2024-09-15T02:49:04.419 INFO:tasks.rados.rados.0.smithi191.stdout:211: snap_remove snap 9 2024-09-15T02:49:04.444 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:04.465 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:04.461+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:04.473 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:04.480 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:04.480 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:04.480 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:04.481 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:04.481 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:04.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:04.511 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:04.550 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:04.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:04.543+0000 7f54d95f9700 0 -- 172.21.15.6:0/2094484730 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7f54d4194070 msgr2=0x7f54d4196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T02:49:04.566 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:04.561+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:04.611 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.7s6 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:49.632655+0000", 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 14.973685335000001, 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011455284, 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.632655+0000", 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.632655+0000", 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.632656+0000", 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.02e-06 2024-09-15T02:49:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.633091+0000", 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043513100000000001 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.633159+0000", 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7776999999999998e-05 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.633176+0000", 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6841999999999999e-05 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.634085+0000", 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090872299999999995 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.634114+0000", 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9189000000000001e-05 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.644082+0000", 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0099678920000000008 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:49.644111+0000", 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8710000000000001e-05 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.956924259000001, 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01495401, 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649418+0000", 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.553e-06 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649682+0000", 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026439200000000001 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649749+0000", 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6585e-05 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649761+0000", 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1584e-05 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649838+0000", 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7866000000000003e-05 2024-09-15T02:49:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649852+0000", 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4e-05 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664337+0000", 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014485031000000001 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664370+0000", 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2999e-05 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.858045685, 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014289939, 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6199999999999999e-07 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748880+0000", 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000584813 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748952+0000", 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1798000000000006e-05 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748962+0000", 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0567999999999999e-05 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.749080+0000", 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117314 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.749096+0000", 2024-09-15T02:49:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6282000000000001e-05 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.762525+0000", 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013428993 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.762585+0000", 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9908999999999998e-05 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.847113626000001, 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015371035, 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759228+0000", 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3599999999999999e-06 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759805+0000", 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057638899999999996 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759906+0000", 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010075499999999999 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759920+0000", 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4166e-05 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.760017+0000", 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7647999999999997e-05 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.760038+0000", 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0290999999999999e-05 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.774535+0000", 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014497252 2024-09-15T02:49:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.774598+0000", 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3174000000000002e-05 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:421 3.cs0 3:319e3fcc:::smithi19125844-50:head [append 0~0] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266390+0000", 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.33995058, 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010849269, 2024-09-15T02:49:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 421 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266390+0000", 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266390+0000", 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266393+0000", 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4880000000000001e-06 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266412+0000", 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8966000000000001e-05 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266435+0000", 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3595e-05 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266446+0000", 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0387999999999999e-05 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266542+0000", 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6903000000000004e-05 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266645+0000", 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102624 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.267589+0000", 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000943472 2024-09-15T02:49:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276888+0000", 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092990469999999995 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277159+0000", 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000271842 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277239+0000", 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9944000000000006e-05 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:422 3.cs0 3:319e3fcc:::smithi19125844-50:head [rollback a] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.339871147, 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028855308, 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 422 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.85e-07 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266475+0000", 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.527e-06 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266498+0000", 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3652999999999999e-05 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266521+0000", 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3006999999999998e-05 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268383+0000", 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018616730000000001 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268454+0000", 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0479000000000001e-05 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275682+0000", 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072287380000000002 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283977+0000", 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082944760000000003 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295223+0000", 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011245890999999999 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295325+0000", 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102079 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:423 3.cs0 3:319e3fcc:::smithi19125844-50:head [append 0~0] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.339801189999999, 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028955272000000001, 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 423 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266541+0000", 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5719999999999999e-06 2024-09-15T02:49:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266548+0000", 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4899999999999997e-06 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266559+0000", 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1158000000000001e-05 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266564+0000", 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1630000000000004e-06 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276125+0000", 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095610759999999999 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276243+0000", 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117991 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276700+0000", 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00045704300000000002 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286454+0000", 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097542040000000007 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295420+0000", 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089656649999999994 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295495+0000", 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4909999999999999e-05 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.331240813000001, 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014524752, 2024-09-15T02:49:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275101+0000", 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6899999999999996e-07 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275587+0000", 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000485976 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275641+0000", 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4728999999999997e-05 2024-09-15T02:49:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275650+0000", 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4749999999999993e-06 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275756+0000", 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105877 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275768+0000", 2024-09-15T02:49:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2734999999999999e-05 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289496+0000", 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013727892 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289625+0000", 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012829900000000001 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.7s6 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.289346+0000", 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.316994292, 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011499304, 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289346+0000", 2024-09-15T02:49:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289346+0000", 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289347+0000", 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0559999999999999e-06 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289674+0000", 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032672499999999999 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289821+0000", 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014660899999999999 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289827+0000", 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5289999999999997e-06 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289885+0000", 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7598e-05 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289911+0000", 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5678999999999999e-05 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.300814+0000", 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010902960999999999 2024-09-15T02:49:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.300846+0000", 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2147000000000002e-05 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.7s6 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.289678+0000", 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.316662215000001, 2024-09-15T02:49:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011181155, 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289678+0000", 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289678+0000", 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289679+0000", 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.99e-07 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289928+0000", 2024-09-15T02:49:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024907199999999998 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.290055+0000", 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012674299999999999 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.290066+0000", 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1496e-05 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.294879+0000", 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048125859999999998 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.294894+0000", 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5364000000000001e-05 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.300851+0000", 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059563430000000002 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.300860+0000", 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.952e-06 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:442 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~0] snapc 10=[10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:55.229833+0000", 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 9.3765072790000001, 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010869953, 2024-09-15T02:49:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 442 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.229833+0000", 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.229833+0000", 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.229834+0000", 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8299999999999995e-07 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.229838+0000", 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0040000000000001e-06 2024-09-15T02:49:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.229851+0000", 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2276000000000001e-05 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.229857+0000", 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9800000000000003e-06 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.234867+0000", 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050104939999999999 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.235013+0000", 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000145649 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.235665+0000", 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065247700000000005 2024-09-15T02:49:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.238085+0000", 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024202049999999999 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.240579+0000", 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024936899999999998 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:55.240703+0000", 2024-09-15T02:49:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012429499999999999 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:512 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~786432] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.362974197, 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026925933999999999, 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 512 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243369+0000", 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7709999999999998e-06 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.245943+0000", 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025742260000000002 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.245980+0000", 2024-09-15T02:49:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6510000000000001e-05 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246008+0000", 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8286e-05 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246085+0000", 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6760000000000004e-05 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246181+0000", 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5546999999999998e-05 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.247292+0000", 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011118090000000001 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261781+0000", 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014488861 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270200+0000", 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084192450000000005 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270292+0000", 2024-09-15T02:49:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1918999999999999e-05 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:513 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~491520] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.360054504, 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024200269, 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 513 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246289+0000", 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7329999999999999e-06 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248021+0000", 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017319410000000001 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248044+0000", 2024-09-15T02:49:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3451999999999998e-05 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248057+0000", 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2745000000000001e-05 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.253501+0000", 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054438999999999998 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.253636+0000", 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000134587 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.254328+0000", 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000692474 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074557210000000002 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270429+0000", 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0086448540000000004 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270486+0000", 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7862000000000003e-05 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.679 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:514 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~278528] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.358213212, 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023758102, 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 514 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.680 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248129+0000", 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4389999999999999e-06 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.681 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248897+0000", 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000768591 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248919+0000", 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1985999999999999e-05 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248931+0000", 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1516e-05 2024-09-15T02:49:04.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.257681+0000", 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087502150000000004 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.257788+0000", 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106774 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.258392+0000", 2024-09-15T02:49:04.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060445900000000001 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266310+0000", 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079177050000000006 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271830+0000", 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055203140000000001 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271886+0000", 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5102999999999997e-05 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:515 3.fs0 3:f0e056ea:::smithi19125844-7:head [setxattr _header (56) in=7b] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.35740028899999998, 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023210873999999999, 2024-09-15T02:49:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 515 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.686 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248941+0000", 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.035e-06 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248945+0000", 2024-09-15T02:49:04.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0330000000000002e-06 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248957+0000", 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1669e-05 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248962+0000", 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2650000000000004e-06 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.260634+0000", 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011671488000000001 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.260737+0000", 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102816 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261347+0000", 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000610043 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266315+0000", 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049678099999999996 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272042+0000", 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005727869 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272151+0000", 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010884599999999999 2024-09-15T02:49:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:516 3.fs0 3:f0e056ea:::smithi19125844-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.35737266099999998, 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.025266506000000001, 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.691 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 516 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248969+0000", 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4700000000000001e-07 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248971+0000", 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.588e-06 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248983+0000", 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1579999999999999e-05 2024-09-15T02:49:04.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248995+0000", 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.185e-05 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261706+0000", 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012710971 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:04.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261774+0000", 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8342000000000004e-05 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272225+0000", 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010450971999999999 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272297+0000", 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1903999999999995e-05 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.695 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.274235+0000", 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019377520000000001 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:520 3.fs0 3:f39d968a:::smithi19125844-6:head [append 0~0] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.35734079400000002, 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024220364000000001, 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.696 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 520 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2100000000000002e-07 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249003+0000", 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4779999999999998e-06 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249010+0000", 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1799999999999999e-06 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249018+0000", 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1300000000000001e-06 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261843+0000", 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012824894999999999 2024-09-15T02:49:04.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262019+0000", 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017557599999999999 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263083+0000", 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001064206 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266322+0000", 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032392810000000001 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273128+0000", 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068061969999999999 2024-09-15T02:49:04.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273220+0000", 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2e-05 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:521 3.fs0 3:f39d968a:::smithi19125844-6:head [rollback 15] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.35731686400000001, 2024-09-15T02:49:04.702 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024414946999999999, 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 521 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3700000000000001e-07 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249027+0000", 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7819999999999998e-06 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249038+0000", 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0569e-05 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249043+0000", 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2129999999999999e-06 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263801+0000", 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014758603 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263989+0000", 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000188004 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.264897+0000", 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090739499999999997 2024-09-15T02:49:04.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.268454+0000", 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035574119999999998 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273370+0000", 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049154689999999996 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273439+0000", 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9062999999999999e-05 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 60/33 ECSubWrite(tid=75, reqid=client.4526.0:529, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.34487837500000001, 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022861161000000001, 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.708 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.709 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261463+0000", 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.13e-06 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261848+0000", 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038425800000000001 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261906+0000", 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7896000000000003e-05 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261913+0000", 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8269999999999998e-06 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261989+0000", 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.559e-05 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262002+0000", 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3291e-05 2024-09-15T02:49:04.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284285+0000", 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022282659999999999 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284323+0000", 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8509e-05 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 60/33 ECSubWrite(tid=76, reqid=client.4526.0:530, at_version=60'53, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.33586811799999999, 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013872888, 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9599999999999999e-07 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:04.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271037+0000", 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00056329999999999998 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271079+0000", 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2446e-05 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271086+0000", 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0439999999999998e-06 2024-09-15T02:49:04.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276187+0000", 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051012849999999997 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276202+0000", 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4887e-05 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:04.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284331+0000", 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081288690000000004 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284345+0000", 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4361e-05 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:04.717 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:04.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:04.719 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:04.713+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:04.783 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:04.785 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:04.785 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:04.785 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:04.786 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:04.787 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:04.820 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:04.818+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:04.866 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:04.866 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:04.866 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:04.866 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:04.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:04.921 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:04.917+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:04.979 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:04.979 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:04.979 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:04.979 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:04.979 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:04.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:05.021 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:05.015+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:05.093 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 38/33 ECSubWrite(tid=19, reqid=osd.15.0:19, at_version=38'14, trim_to=0'0, roll_forward_to=37'12))", 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:42.534265+0000", 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.554696016000001, 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010812166999999999, 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534265+0000", 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534265+0000", 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534266+0000", 2024-09-15T02:49:05.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534315+0000", 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8541000000000002e-05 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534422+0000", 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107373 2024-09-15T02:49:05.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534438+0000", 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5906e-05 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.536647+0000", 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002209262 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.536661+0000", 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4192000000000001e-05 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.545032+0000", 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083703709999999997 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.545078+0000", 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5948000000000001e-05 2024-09-15T02:49:05.098 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.520296044999998, 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016202787999999999, 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568666+0000", 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0699999999999997e-07 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569158+0000", 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000491633 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569218+0000", 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0742999999999998e-05 2024-09-15T02:49:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569231+0000", 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2455e-05 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569315+0000", 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3701000000000004e-05 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569329+0000", 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4606e-05 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.584839+0000", 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015510222000000001 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.584868+0000", 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8821000000000001e-05 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.103 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.104 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.104 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 14.439347486000001, 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014988552, 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:05.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649615+0000", 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8800000000000002e-07 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650002+0000", 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038781599999999997 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650043+0000", 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0312999999999998e-05 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650051+0000", 2024-09-15T02:49:05.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2390000000000002e-06 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650118+0000", 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7015999999999997e-05 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650127+0000", 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6200000000000005e-06 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664560+0000", 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014433692 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664603+0000", 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2268000000000003e-05 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.109 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.397038734000001, 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013640826, 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691924+0000", 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.544e-06 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692182+0000", 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025800999999999998 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692284+0000", 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010141699999999999 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692295+0000", 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1447999999999999e-05 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692382+0000", 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7294999999999997e-05 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692401+0000", 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8635e-05 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.705531+0000", 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013129403 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.705564+0000", 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3074000000000002e-05 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=160, reqid=client.4526.0:364, at_version=49'23, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.391747459999999, 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011544153, 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.114 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697215+0000", 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.046e-06 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697510+0000", 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00029446300000000002 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697550+0000", 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0247999999999998e-05 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697562+0000", 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2068e-05 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.699512+0000", 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019506219999999999 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.699531+0000", 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8343999999999999e-05 2024-09-15T02:49:05.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708741+0000", 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092103159999999996 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708758+0000", 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7045999999999999e-05 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=173, reqid=client.4526.0:382, at_version=49'30, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.732799+0000", 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.356162952, 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010449173000000001, 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732799+0000", 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732799+0000", 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732799+0000", 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3399999999999999e-07 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732814+0000", 2024-09-15T02:49:05.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4443e-05 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732903+0000", 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9048999999999994e-05 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.732908+0000", 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2000000000000002e-06 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.740358+0000", 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074502300000000004 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.740367+0000", 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4419999999999998e-06 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.743204+0000", 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028370940000000001 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.743248+0000", 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4081e-05 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.759279+0000", 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.329682261, 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010198473, 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.123 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759279+0000", 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759279+0000", 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759281+0000", 2024-09-15T02:49:05.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.353e-06 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759734+0000", 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00045293299999999998 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759791+0000", 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7559999999999998e-05 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759806+0000", 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5378000000000001e-05 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759914+0000", 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010726900000000001 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759956+0000", 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2382000000000002e-05 2024-09-15T02:49:05.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.769443+0000", 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0094864649999999995 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.769478+0000", 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5132999999999997e-05 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 49/33 ECSubWrite(tid=48, reqid=client.4526.0:404, at_version=49'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.769504+0000", 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.319457101999999, 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092127140000000003, 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.769504+0000", 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.769504+0000", 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.769505+0000", 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.770034+0000", 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052825500000000004 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.770088+0000", 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4181000000000002e-05 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.770094+0000", 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6989999999999997e-06 2024-09-15T02:49:05.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.770152+0000", 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7608999999999997e-05 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.770163+0000", 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1387e-05 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.778690+0000", 2024-09-15T02:49:05.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0085268749999999997 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.778717+0000", 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6786000000000001e-05 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:05.132 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.835388539, 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031206293, 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254463+0000", 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088986800000000004 2024-09-15T02:49:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254519+0000", 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5640000000000003e-05 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254529+0000", 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0152999999999999e-05 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254650+0000", 2024-09-15T02:49:05.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012129300000000001 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254679+0000", 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9107000000000001e-05 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284735+0000", 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.030055693000000001 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284779+0000", 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4199000000000002e-05 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.828495304, 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024332761000000001, 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260468+0000", 2024-09-15T02:49:05.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.508e-06 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260940+0000", 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047216899999999998 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.261103+0000", 2024-09-15T02:49:05.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016333 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.261116+0000", 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.327e-05 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.269681+0000", 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0085646730000000001 2024-09-15T02:49:05.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.269691+0000", 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4339999999999998e-06 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284787+0000", 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015096058000000001 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284799+0000", 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2319e-05 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.820913082000001, 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01876924, 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268049+0000", 2024-09-15T02:49:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268412+0000", 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000363111 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268461+0000", 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9150000000000002e-05 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268470+0000", 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5909999999999996e-06 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268578+0000", 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107676 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268591+0000", 2024-09-15T02:49:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3733e-05 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286797+0000", 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.018205193000000001 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286818+0000", 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1012000000000001e-05 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=195, reqid=client.4526.0:411, at_version=50'34, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.818844506, 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014722531000000001, 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270118+0000", 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.012e-06 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270182+0000", 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4114000000000003e-05 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270229+0000", 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6473e-05 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270241+0000", 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2092999999999999e-05 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275025+0000", 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047846529999999998 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275036+0000", 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0637e-05 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284820+0000", 2024-09-15T02:49:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097838739999999997 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284840+0000", 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9675000000000001e-05 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.813549479000001, 2024-09-15T02:49:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014093722, 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275413+0000", 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1459999999999999e-06 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277178+0000", 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017643979999999999 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277259+0000", 2024-09-15T02:49:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1926000000000001e-05 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277268+0000", 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4249999999999998e-06 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277340+0000", 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2126999999999996e-05 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277360+0000", 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9678e-05 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289453+0000", 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012093051 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289506+0000", 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2970999999999998e-05 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=196, reqid=client.4526.0:412, at_version=50'35, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275559+0000", 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.813402136000001, 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011112530000000001, 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275559+0000", 2024-09-15T02:49:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275559+0000", 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275588+0000", 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8394999999999998e-05 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275613+0000", 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.563e-05 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275663+0000", 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9629999999999997e-05 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275671+0000", 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2160000000000006e-06 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283502+0000", 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078303230000000001 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283518+0000", 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6617999999999999e-05 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286646+0000", 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031278539999999998 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286672+0000", 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5863999999999998e-05 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.277911+0000", 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.811050542, 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011518654999999999, 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.160 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277911+0000", 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277911+0000", 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277912+0000", 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.066e-06 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278153+0000", 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024120700000000001 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278215+0000", 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1956000000000001e-05 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278225+0000", 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0292000000000001e-05 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278314+0000", 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8991000000000002e-05 2024-09-15T02:49:05.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278325+0000", 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0584000000000001e-05 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289387+0000", 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011061903 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289430+0000", 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2656e-05 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as1 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.927014+0000", 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.1619476000000004, 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095413809999999998, 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927014+0000", 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927014+0000", 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927016+0000", 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.12e-06 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927170+0000", 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000153633 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927289+0000", 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118861 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927309+0000", 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.09e-05 2024-09-15T02:49:05.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927460+0000", 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015032000000000001 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927491+0000", 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1616999999999999e-05 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936458+0000", 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089669729999999996 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936555+0000", 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6957e-05 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as1 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:05.169 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 3.7768353690000001, 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012325150999999999, 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:05.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312128+0000", 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.59e-06 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312511+0000", 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038326000000000001 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312619+0000", 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010794 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312637+0000", 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8104000000000001e-05 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312798+0000", 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016069599999999999 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312831+0000", 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3179999999999997e-05 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324419+0000", 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011588022999999999 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324451+0000", 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2357999999999997e-05 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 60/33 ECSubWrite(tid=75, reqid=client.4526.0:529, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.82705314900000004, 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014182907999999999, 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.174 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261909+0000", 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262920+0000", 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001011236 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262962+0000", 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1980999999999997e-05 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262988+0000", 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6571e-05 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263089+0000", 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100937 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263102+0000", 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3183999999999999e-05 2024-09-15T02:49:05.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276063+0000", 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012960833 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276091+0000", 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8002999999999999e-05 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 60/33 ECSubWrite(tid=76, reqid=client.4526.0:530, at_version=60'53, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.270513+0000", 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.818448222, 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097392120000000006, 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270513+0000", 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270513+0000", 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270514+0000", 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3600000000000002e-07 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270916+0000", 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00040161400000000001 2024-09-15T02:49:05.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270962+0000", 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5847999999999999e-05 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270970+0000", 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9240000000000007e-06 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271052+0000", 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2256999999999995e-05 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271064+0000", 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2036e-05 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.280220+0000", 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091562269999999994 2024-09-15T02:49:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.280252+0000", 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2369999999999997e-05 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.413690+0000", 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.67527198799999999, 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0093977209999999995, 2024-09-15T02:49:05.183 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413690+0000", 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413690+0000", 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413690+0000", 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5600000000000002e-07 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413819+0000", 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012877200000000001 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413890+0000", 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1565999999999999e-05 2024-09-15T02:49:05.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413902+0000", 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1613e-05 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413998+0000", 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6237999999999996e-05 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414021+0000", 2024-09-15T02:49:05.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2884e-05 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423051+0000", 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090300490000000001 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423087+0000", 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6343000000000001e-05 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.188 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:05.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:05.190 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:05.185+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:05.290 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:05.286+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:211: done (3 left) 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:212: delete oid 48 current snap is 15 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:210: finishing rollback tid 0 to smithi19125844-3 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:210: finishing rollback tid 1 to smithi19125844-3 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:209: finishing write tid 1 to smithi19125844-42 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:209: finishing write tid 2 to smithi19125844-42 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:210: finishing rollback tid 2 to smithi19125844-3 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 3 v 33 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:209: finishing write tid 3 to smithi19125844-42 2024-09-15T02:49:05.327 INFO:tasks.rados.rados.0.smithi191.stdout:209: finishing write tid 4 to smithi19125844-42 2024-09-15T02:49:05.328 INFO:tasks.rados.rados.0.smithi191.stdout:209: finishing write tid 5 to smithi19125844-42 2024-09-15T02:49:05.328 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 42 v 9 (ObjNum 96 snap 15 seq_num 96) dirty exists 2024-09-15T02:49:05.328 INFO:tasks.rados.rados.0.smithi191.stdout:209: left oid 42 (ObjNum 96 snap 15 seq_num 96) 2024-09-15T02:49:05.328 INFO:tasks.rados.rados.0.smithi191.stdout:203: finishing rollback tid 2 to smithi19125844-6 2024-09-15T02:49:05.328 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 6 v 31 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T02:49:05.335 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:49:05.343 INFO:tasks.rados.rados.0.smithi191.stdout:203: done (3 left) 2024-09-15T02:49:05.343 INFO:tasks.rados.rados.0.smithi191.stdout:209: done (2 left) 2024-09-15T02:49:05.343 INFO:tasks.rados.rados.0.smithi191.stdout:210: done (1 left) 2024-09-15T02:49:05.343 INFO:tasks.rados.rados.0.smithi191.stdout:212: done (0 left) 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:213: read oid 7 snap 13 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:213: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:214: read oid 40 snap -1 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:214: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:215: rollback oid 30 current snap is 15 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 30 to 13 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 44 current snap is 15 2024-09-15T02:49:05.344 INFO:tasks.rados.rados.0.smithi191.stdout:216: seq_num 98 ranges {737280=491520,1228800=589824,1818624=557056,2375680=16384} 2024-09-15T02:49:05.351 INFO:tasks.rados.rados.0.smithi191.stdout:216: writing smithi19125844-44 from 737280 to 1228800 tid 1 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 61/33 ECSubWrite(tid=62, reqid=osd.12.0:60, at_version=0'0, trim_to=0'0, roll_forward_to=61'11))", 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.351074+0000", 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.00015456000000000001, 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017858899999999999, 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351074+0000", 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351074+0000", 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351075+0000", 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0580000000000001e-06 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351091+0000", 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5676e-05 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351141+0000", 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0454000000000002e-05 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351149+0000", 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8620000000000003e-06 2024-09-15T02:49:05.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351204+0000", 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4293999999999999e-05 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.351214+0000", 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0719e-05 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.357 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-15T02:49:05.357 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 1 2024-09-15T02:49:05.357 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:05.357 INFO:tasks.rados.rados.0.smithi191.stdout:216: writing smithi19125844-44 from 1228800 to 1818624 tid 2 2024-09-15T02:49:05.360 INFO:tasks.rados.rados.0.smithi191.stdout:216: writing smithi19125844-44 from 1818624 to 2375680 tid 3 2024-09-15T02:49:05.360 INFO:tasks.rados.rados.0.smithi191.stdout:216: writing smithi19125844-44 from 2375680 to 2392064 tid 4 2024-09-15T02:49:05.360 INFO:tasks.rados.rados.0.smithi191.stdout:217: delete oid 13 current snap is 15 2024-09-15T02:49:05.360 INFO:tasks.rados.rados.0.smithi191.stdout:213: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-15T02:49:05.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:05.365 INFO:tasks.rados.rados.0.smithi191.stdout:214: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-15T02:49:05.385 INFO:tasks.rados.rados.0.smithi191.stdout:215: finishing rollback tid 0 to smithi19125844-30 2024-09-15T02:49:05.385 INFO:tasks.rados.rados.0.smithi191.stdout:215: finishing rollback tid 1 to smithi19125844-30 2024-09-15T02:49:05.385 INFO:tasks.rados.rados.0.smithi191.stdout:215: finishing rollback tid 2 to smithi19125844-30 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 30 v 44 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: finishing write tid 1 to smithi19125844-44 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: finishing write tid 2 to smithi19125844-44 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: finishing write tid 3 to smithi19125844-44 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: finishing write tid 4 to smithi19125844-44 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: finishing write tid 5 to smithi19125844-44 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: finishing write tid 6 to smithi19125844-44 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 44 v 34 (ObjNum 98 snap 15 seq_num 98) dirty exists 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:216: left oid 44 (ObjNum 98 snap 15 seq_num 98) 2024-09-15T02:49:05.386 INFO:tasks.rados.rados.0.smithi191.stdout:213: done (4 left) 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:214: done (3 left) 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:215: done (2 left) 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:216: done (1 left) 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:217: done (0 left) 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:218: setattr oid 12 current snap is 15 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:219: read oid 1 snap -1 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:219: expect (ObjNum 82 snap 9 seq_num 82) 2024-09-15T02:49:05.387 INFO:tasks.rados.rados.0.smithi191.stdout:220: snap_remove snap 13 2024-09-15T02:49:05.390 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:05.386+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:05.458 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:05.458 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:05.458 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:05.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:05.492 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:05.489+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:05.568 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:231 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.998554828, 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012963349000000001, 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.569 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 231 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565064+0000", 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1140000000000002e-06 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565078+0000", 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3900000000000001e-05 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565120+0000", 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2070999999999998e-05 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565135+0000", 2024-09-15T02:49:05.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5085999999999999e-05 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565263+0000", 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012826600000000001 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565272+0000", 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0259999999999996e-06 2024-09-15T02:49:05.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569405+0000", 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041323439999999996 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569560+0000", 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015537600000000001 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.570911+0000", 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013509349999999999 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574145+0000", 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003233685 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:05.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.577972+0000", 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038275480000000001 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578024+0000", 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1997999999999999e-05 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:232 3.5s0 3:a906e5df:::smithi19125844-36:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.998450869999999, 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013333868, 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.577 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 232 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565166+0000", 2024-09-15T02:49:05.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.054e-06 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565170+0000", 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1969999999999998e-06 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565205+0000", 2024-09-15T02:49:05.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4829999999999997e-05 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565234+0000", 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9320999999999999e-05 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565392+0000", 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015784999999999999 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565408+0000", 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5434000000000002e-05 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571539+0000", 2024-09-15T02:49:05.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061311990000000004 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571658+0000", 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119486 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572348+0000", 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068956300000000002 2024-09-15T02:49:05.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574147+0000", 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017987459999999999 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578458+0000", 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043109690000000004 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578499+0000", 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1219e-05 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:233 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.998370877999999, 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.49017684700000003, 2024-09-15T02:49:05.585 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 233 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565246+0000", 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565250+0000", 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.76e-06 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565261+0000", 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1693e-05 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565268+0000", 2024-09-15T02:49:05.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3570000000000003e-06 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565481+0000", 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021371699999999999 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565537+0000", 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5838000000000003e-05 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572781+0000", 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072440639999999997 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572928+0000", 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014673400000000001 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.573788+0000", 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00086026199999999998 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582219+0000", 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084309350000000005 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055303+0000", 2024-09-15T02:49:05.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.47308344099999999 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055422+0000", 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119119 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.592 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/33 ECSubWrite(tid=23, reqid=client.4526.0:236, at_version=39'18, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.989545273000001, 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089568029999999993, 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6399999999999998e-07 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.594 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574843+0000", 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00077236399999999999 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575043+0000", 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019989399999999999 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575050+0000", 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.528e-06 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575124+0000", 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4041999999999993e-05 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575134+0000", 2024-09-15T02:49:05.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582994+0000", 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078592579999999992 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.583027+0000", 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3747000000000001e-05 2024-09-15T02:49:05.597 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 39/33 ECSubWriteReply(tid=18, last_complete=39'11, committed=1, applied=1))", 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.987359072, 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.480047051, 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576262+0000", 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.074e-06 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055023+0000", 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.478760992 2024-09-15T02:49:05.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055037+0000", 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3607e-05 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055156+0000", 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118522 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.056304+0000", 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001148154 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:249 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~704512] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.977975614000002, 2024-09-15T02:49:05.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031968222999999997, 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:05.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585643+0000", 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7839999999999999e-06 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587905+0000", 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022617729999999999 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587945+0000", 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.981e-05 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587959+0000", 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4297000000000001e-05 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588041+0000", 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2514000000000003e-05 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588163+0000", 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012119 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589878+0000", 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017156809999999999 2024-09-15T02:49:05.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607051+0000", 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017173134999999999 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617495+0000", 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010443661999999999 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617608+0000", 2024-09-15T02:49:05.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113377 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:250 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~376832] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.974697503000002, 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028844499999999999, 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.669 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 250 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588920+0000", 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.807e-06 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589853+0000", 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00093252000000000005 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589887+0000", 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4721000000000002e-05 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589910+0000", 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2787000000000001e-05 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601237+0000", 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011326924 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601404+0000", 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016697000000000001 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.602323+0000", 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00091864900000000001 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607163+0000", 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048403409999999997 2024-09-15T02:49:05.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617711+0000", 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010548119999999999 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617763+0000", 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1660999999999999e-05 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:307 3.5s0 3:ae24c343:::smithi19125844-3:head [copy-from ver 10] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 14.944387196999999, 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.021447352999999999, 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 307 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619232+0000", 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619263+0000", 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1090000000000002e-05 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619289+0000", 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5284000000000001e-05 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619300+0000", 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1518000000000001e-05 2024-09-15T02:49:05.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619367+0000", 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6983000000000006e-05 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619450+0000", 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2785000000000001e-05 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.630087+0000", 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010637047 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635664+0000", 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055774500000000003 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640607+0000", 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049425019999999997 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640676+0000", 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9078000000000005e-05 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:308 3.5s0 3:ae24c343:::smithi19125844-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 14.944297560999999, 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016310021000000001, 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 308 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.681 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619319+0000", 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9500000000000002e-07 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619324+0000", 2024-09-15T02:49:05.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.87e-06 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619332+0000", 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2539999999999992e-06 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619335+0000", 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0079999999999998e-06 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619592+0000", 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000256724 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619634+0000", 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2011999999999997e-05 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635446+0000", 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015811935999999999 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635533+0000", 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7126000000000006e-05 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635628+0000", 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5495999999999995e-05 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:05.685 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 14.913525587000001, 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013723521000000001, 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650091+0000", 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04e-06 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650742+0000", 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000650436 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650792+0000", 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0680000000000003e-05 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650812+0000", 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9397000000000001e-05 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650905+0000", 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3294000000000005e-05 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650922+0000", 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7209000000000001e-05 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663784+0000", 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012861490999999999 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663814+0000", 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9974000000000001e-05 2024-09-15T02:49:05.689 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 49/33 ECSubWrite(tid=175, reqid=client.4526.0:354, at_version=49'27, trim_to=0'0, roll_forward_to=46'25))", 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.667878+0000", 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.895737769, 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064772570000000002, 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.690 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.667878+0000", 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.667878+0000", 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.667879+0000", 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0299999999999999e-07 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.691 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668189+0000", 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000310582 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668318+0000", 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012915100000000001 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668339+0000", 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0990000000000001e-05 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668545+0000", 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000205995 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.668575+0000", 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0253999999999999e-05 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.674297+0000", 2024-09-15T02:49:05.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057217830000000003 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.674355+0000", 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7998999999999998e-05 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:05.694 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.861845665000001, 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084791039999999995, 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:05.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5699999999999999e-07 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701896+0000", 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012543400000000001 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.696 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701967+0000", 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1172000000000002e-05 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701978+0000", 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1e-05 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702056+0000", 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7615999999999997e-05 2024-09-15T02:49:05.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702069+0000", 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3746e-05 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710220+0000", 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081502959999999996 2024-09-15T02:49:05.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710249+0000", 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9583000000000001e-05 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.288198524, 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092230960000000001, 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.701 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3300000000000001e-07 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.702 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276746+0000", 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013288589999999999 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276893+0000", 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000146849 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276906+0000", 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2347000000000001e-05 2024-09-15T02:49:05.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276979+0000", 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3198e-05 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277005+0000", 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5837e-05 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284582+0000", 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075769779999999998 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284640+0000", 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8795000000000002e-05 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.636377328, 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009421937, 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927241+0000", 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0899999999999999e-06 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927476+0000", 2024-09-15T02:49:05.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000235047 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927608+0000", 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013218999999999999 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927627+0000", 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.965e-05 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927797+0000", 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016982 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927817+0000", 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9939999999999999e-05 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936590+0000", 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087725600000000004 2024-09-15T02:49:05.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936660+0000", 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0640000000000001e-05 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:05.710 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.2508760179999996, 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014263519000000001, 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312741+0000", 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1999999999999998e-07 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313098+0000", 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035754 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313251+0000", 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000153177 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313275+0000", 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3365000000000001e-05 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313599+0000", 2024-09-15T02:49:05.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032383899999999997 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313628+0000", 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.921e-05 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326410+0000", 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012781957 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.327003+0000", 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00059361099999999997 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 57/33 ECSubWrite(tid=39, reqid=client.4526.0:482, at_version=57'24, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.2346277780000001, 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010144892000000001, 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328990+0000", 2024-09-15T02:49:05.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7099999999999999e-06 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329132+0000", 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000142068 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329222+0000", 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0607999999999997e-05 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.717 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329240+0000", 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7524e-05 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329313+0000", 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2658999999999997e-05 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329323+0000", 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0497e-05 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339071+0000", 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097483840000000006 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339133+0000", 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1441999999999999e-05 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.149790106, 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096704219999999997, 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7800000000000001e-07 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413927+0000", 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101127 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.721 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413988+0000", 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1103000000000001e-05 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413997+0000", 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7709999999999995e-06 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414099+0000", 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102274 2024-09-15T02:49:05.722 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414115+0000", 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6223e-05 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423433+0000", 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0093179969999999997 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423496+0000", 2024-09-15T02:49:05.723 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2749000000000005e-05 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/33 ECSubWrite(tid=54, reqid=client.4526.0:537, at_version=60'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.424373+0000", 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.1392428960000001, 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080799630000000008, 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.724 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424373+0000", 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424373+0000", 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424374+0000", 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.127e-06 2024-09-15T02:49:05.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424544+0000", 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017042399999999999 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424607+0000", 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2649000000000003e-05 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424614+0000", 2024-09-15T02:49:05.726 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8109999999999997e-06 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424739+0000", 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012517600000000001 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424749+0000", 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0293e-05 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.727 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432395+0000", 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076458740000000004 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432453+0000", 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7608999999999997e-05 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/33 ECSubWrite(tid=55, reqid=client.4526.0:538, at_version=60'8, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.432747+0000", 2024-09-15T02:49:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.1308692469999999, 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072342810000000004, 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432747+0000", 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432747+0000", 2024-09-15T02:49:05.729 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432748+0000", 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1030000000000001e-06 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432828+0000", 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0646e-05 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432895+0000", 2024-09-15T02:49:05.730 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6482000000000005e-05 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432902+0000", 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6920000000000002e-06 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432966+0000", 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3879000000000003e-05 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.731 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432980+0000", 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3617e-05 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.439953+0000", 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069725689999999996 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.439981+0000", 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8293000000000002e-05 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.732 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 61/33 ECSubWrite(tid=61, reqid=osd.12.0:61, at_version=61'38, trim_to=0'0, roll_forward_to=59'34))", 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.344158+0000", 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.219457651, 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060932740000000001, 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.344158+0000", 2024-09-15T02:49:05.733 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.344158+0000", 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.344159+0000", 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4000000000000001e-07 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.734 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.344191+0000", 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.344233+0000", 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1853000000000001e-05 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.344281+0000", 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8025999999999997e-05 2024-09-15T02:49:05.735 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.345125+0000", 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00084408699999999998 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.345135+0000", 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9890000000000004e-06 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:05.736 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.350222+0000", 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050865939999999998 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.350251+0000", 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9856999999999999e-05 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:05.737 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:05.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:05.740 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:05.734+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:05.840 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:05.835+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:05.849 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:05.849 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:05.849 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:05.850 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:05.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:05.941 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:05.937+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:05.962 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:05.962 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:05.962 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:05.963 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:05.963 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:05.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:06.042 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:06.037+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.568653+0000", 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.504727054, 2024-09-15T02:49:06.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013097361, 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568653+0000", 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568653+0000", 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568654+0000", 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2399999999999996e-07 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569309+0000", 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065497999999999999 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569378+0000", 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9105999999999999e-05 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569390+0000", 2024-09-15T02:49:06.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.171e-05 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569510+0000", 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012006399999999999 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569534+0000", 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4337000000000001e-05 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.581715+0000", 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012180471999999999 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.581750+0000", 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5768e-05 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:360 3.as0 3:54c1cad6:::smithi19125844-28:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:49:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.397396247, 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034523553999999998, 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 360 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:49:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675987+0000", 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0900000000000001e-06 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676003+0000", 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6407000000000001e-05 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676045+0000", 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1390000000000002e-05 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676065+0000", 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0157e-05 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676146+0000", 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0891999999999997e-05 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676153+0000", 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0079999999999996e-06 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.678914+0000", 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002761262 2024-09-15T02:49:06.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679005+0000", 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0648000000000004e-05 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.693685+0000", 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014680723 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702650+0000", 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089650400000000005 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710453+0000", 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078028220000000001 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710507+0000", 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4114999999999999e-05 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:361 3.as0 3:54c1cad6:::smithi19125844-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.397282666000001, 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026448546, 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.090 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 361 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676099+0000", 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.72e-06 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676105+0000", 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3729999999999998e-06 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676128+0000", 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2096999999999999e-05 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676139+0000", 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1603999999999999e-05 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676214+0000", 2024-09-15T02:49:06.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4380000000000003e-05 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676221+0000", 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8970000000000008e-06 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679152+0000", 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029307930000000001 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:06.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679218+0000", 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5701e-05 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702293+0000", 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.02307476 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702424+0000", 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013107900000000001 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702546+0000", 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000122142 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.819516413000001, 2024-09-15T02:49:06.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.02916968, 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254277+0000", 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00041306699999999998 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254344+0000", 2024-09-15T02:49:06.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6806999999999999e-05 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254359+0000", 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4738e-05 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254477+0000", 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118509 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254495+0000", 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7297999999999999e-05 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.282999+0000", 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028504345 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283033+0000", 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4436e-05 2024-09-15T02:49:06.100 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.812794558, 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022480231, 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260587+0000", 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7400000000000001e-06 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260827+0000", 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000239459 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260907+0000", 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9956999999999994e-05 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260923+0000", 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6592e-05 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268606+0000", 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076824390000000001 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268615+0000", 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9600000000000006e-06 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283052+0000", 2024-09-15T02:49:06.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014437815 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283066+0000", 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3269e-05 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=195, reqid=client.4526.0:411, at_version=50'34, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.270958+0000", 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.802421771000001, 2024-09-15T02:49:06.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012122575, 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270958+0000", 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270958+0000", 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270959+0000", 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0139999999999999e-06 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.271081+0000", 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012218600000000001 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.271126+0000", 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4116000000000002e-05 2024-09-15T02:49:06.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.271133+0000", 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4089999999999997e-06 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275065+0000", 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039322519999999998 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275097+0000", 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2125000000000002e-05 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283070+0000", 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079729540000000008 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283081+0000", 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0519e-05 2024-09-15T02:49:06.109 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s6 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275529+0000", 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.797850737999999, 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011397446, 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275529+0000", 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275529+0000", 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275530+0000", 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0499999999999999e-07 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276493+0000", 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00096330700000000005 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276670+0000", 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017730099999999999 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276677+0000", 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1300000000000003e-06 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276782+0000", 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010467900000000001 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0854e-05 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286894+0000", 2024-09-15T02:49:06.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010100573 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286927+0000", 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3197000000000001e-05 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:472 3.as0 3:54c1cad6:::smithi19125844-28:head [copy-from ver 18 in=55b] snapc 13=[13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7522750800000004, 2024-09-15T02:49:06.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.61688121200000001, 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 472 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321107+0000", 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9740000000000001e-06 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321117+0000", 2024-09-15T02:49:06.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0214000000000001e-05 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321173+0000", 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5482000000000002e-05 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321185+0000", 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2028000000000001e-05 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321271+0000", 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6592e-05 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321500+0000", 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022896 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927006+0000", 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.60550546299999997 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936994+0000", 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009987915 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.937732+0000", 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073820699999999995 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.937986+0000", 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025437700000000002 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:473 3.as0 3:54c1cad6:::smithi19125844-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7521753020000004, 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.61562847600000004, 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.120 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 473 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321206+0000", 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2440000000000001e-06 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321211+0000", 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4580000000000001e-06 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321237+0000", 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5111999999999999e-05 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321246+0000", 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1519999999999998e-06 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321773+0000", 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052693800000000002 2024-09-15T02:49:06.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321843+0000", 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0831e-05 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936316+0000", 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.61447244099999998 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936584+0000", 2024-09-15T02:49:06.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026809300000000001 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936833+0000", 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024920699999999999 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:481 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~753664] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.307419+0000", 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7659611579999996, 2024-09-15T02:49:06.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.029233761, 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 481 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.307419+0000", 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.307419+0000", 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.307422+0000", 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7930000000000002e-06 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.308924+0000", 2024-09-15T02:49:06.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015024140000000001 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.308980+0000", 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6239999999999997e-05 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.309005+0000", 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4995999999999999e-05 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.309157+0000", 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000151197 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.309361+0000", 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000203983 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312775+0000", 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034144829999999998 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336331+0000", 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023556128999999999 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336551+0000", 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000219865 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336653+0000", 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101661 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.130 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:482 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~720896] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312861+0000", 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7605189399999999, 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027088476, 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 482 2024-09-15T02:49:06.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312861+0000", 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312861+0000", 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312863+0000", 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2500000000000001e-06 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314314+0000", 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014503840000000001 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314363+0000", 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9240000000000003e-05 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314382+0000", 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8919e-05 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326033+0000", 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011651013 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326280+0000", 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024667800000000001 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.327782+0000", 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015028019999999999 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336793+0000", 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090105910000000001 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339895+0000", 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031024189999999999 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339950+0000", 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.418e-05 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:483 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~49152] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.314412+0000", 2024-09-15T02:49:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7589675800000002, 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028623451000000001, 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 483 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314412+0000", 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314412+0000", 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314415+0000", 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3029999999999998e-06 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314530+0000", 2024-09-15T02:49:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115097 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314549+0000", 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9020000000000001e-05 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314559+0000", 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6099999999999995e-06 2024-09-15T02:49:06.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336836+0000", 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022277081000000001 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336930+0000", 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4847999999999997e-05 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.337410+0000", 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047964500000000001 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339004+0000", 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015934930000000001 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.342886+0000", 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038823569999999999 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343036+0000", 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014999699999999999 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:484 3.as0 3:54c1cad6:::smithi19125844-28:head [setxattr _header (57) in=7b] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.314568+0000", 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7588120900000002, 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028727822, 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 484 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314568+0000", 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314568+0000", 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314569+0000", 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.393e-06 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314574+0000", 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.454e-06 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314583+0000", 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0559999999999994e-06 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314588+0000", 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6200000000000004e-06 2024-09-15T02:49:06.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338105+0000", 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023516358000000001 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338219+0000", 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011388899999999999 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338599+0000", 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038068599999999998 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339311+0000", 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000711672 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343168+0000", 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038569210000000001 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343296+0000", 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127773 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:485 3.as0 3:54c1cad6:::smithi19125844-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.314594+0000", 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7587860470000001, 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034676314, 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:06.148 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 485 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314594+0000", 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314594+0000", 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314595+0000", 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7999999999999993e-07 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314598+0000", 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1300000000000001e-06 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314608+0000", 2024-09-15T02:49:06.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4299999999999995e-06 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314613+0000", 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.57e-06 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338862+0000", 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024249297 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338991+0000", 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012851799999999999 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343772+0000", 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047815399999999999 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343870+0000", 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7619000000000002e-05 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.349270+0000", 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054002299999999998 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.153 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.320551+0000", 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7528290719999999, 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015703106000000001, 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320551+0000", 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320551+0000", 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320552+0000", 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3120000000000001e-06 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320564+0000", 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1736000000000001e-05 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320615+0000", 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0899999999999997e-05 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320626+0000", 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0772e-05 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336239+0000", 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015612862 2024-09-15T02:49:06.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336254+0000", 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5523999999999999e-05 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.320765+0000", 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7526153149999999, 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015630415000000002, 2024-09-15T02:49:06.157 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320765+0000", 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320765+0000", 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320766+0000", 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7499999999999998e-07 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320772+0000", 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9070000000000004e-06 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320816+0000", 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4338000000000001e-05 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320827+0000", 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0915e-05 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336384+0000", 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015557034000000001 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336395+0000", 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1246e-05 2024-09-15T02:49:06.160 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.320888+0000", 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7524919250000002, 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01545114, 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320888+0000", 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320888+0000", 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320889+0000", 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0599999999999999e-07 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320896+0000", 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2910000000000002e-06 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320936+0000", 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0083000000000002e-05 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320944+0000", 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0120000000000005e-06 2024-09-15T02:49:06.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336290+0000", 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01534511 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336339+0000", 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9737999999999997e-05 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.321693+0000", 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7516865939999997, 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014666235, 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.321693+0000", 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.321693+0000", 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.321695+0000", 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.046e-06 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.321702+0000", 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9119999999999998e-06 2024-09-15T02:49:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.321751+0000", 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8928000000000003e-05 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.321762+0000", 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0212e-05 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336352+0000", 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014590374 2024-09-15T02:49:06.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336360+0000", 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.763e-06 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.324546+0000", 2024-09-15T02:49:06.168 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.7488343750000004, 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011782269999999999, 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324546+0000", 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324546+0000", 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324547+0000", 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6339999999999999e-06 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324559+0000", 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1932e-05 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324648+0000", 2024-09-15T02:49:06.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9221999999999994e-05 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324662+0000", 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3475999999999999e-05 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336319+0000", 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011657399000000001 2024-09-15T02:49:06.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336328+0000", 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6070000000000008e-06 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 61/33 ECSubWriteReply(tid=62, last_complete=61'11, committed=1, applied=1))", 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:06.172 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.71971348700000004, 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.379230122, 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353672+0000", 2024-09-15T02:49:06.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5719999999999996e-06 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732611+0000", 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.37893902499999998 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732624+0000", 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3166000000000001e-05 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:06.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732771+0000", 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014647299999999999 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732897+0000", 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012614000000000001 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:06.176 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:06.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:06.179 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:06.177+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:06.279 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:06.275+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:06.290 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:06.290 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:06.290 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:06.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:06.333 INFO:tasks.rados.rados.0.smithi191.stdout:220: done (2 left) 2024-09-15T02:49:06.333 INFO:tasks.rados.rados.0.smithi191.stdout:221: read oid 40 snap -1 2024-09-15T02:49:06.333 INFO:tasks.rados.rados.0.smithi191.stdout:221: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-15T02:49:06.333 INFO:tasks.rados.rados.0.smithi191.stdout:222: snap_create 2024-09-15T02:49:06.333 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 12 v 63 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T02:49:06.333 INFO:tasks.rados.rados.0.smithi191.stdout:219: expect (ObjNum 82 snap 9 seq_num 82) 2024-09-15T02:49:06.380 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:06.375+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:06.410 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:06.411 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:06.411 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:06.411 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:06.411 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:06.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:06.481 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:06.477+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:06.506 INFO:tasks.rados.rados.0.smithi191.stdout:221: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-15T02:49:06.515 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.934860739000001, 2024-09-15T02:49:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010162803999999999, 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:49:06.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577824+0000", 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577825+0000", 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9299999999999996e-07 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578183+0000", 2024-09-15T02:49:06.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035738600000000001 2024-09-15T02:49:06.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578258+0000", 2024-09-15T02:49:06.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5891999999999998e-05 2024-09-15T02:49:06.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578278+0000", 2024-09-15T02:49:06.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9641999999999999e-05 2024-09-15T02:49:06.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578517+0000", 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000238619 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578582+0000", 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5091000000000004e-05 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.587957+0000", 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0093750499999999994 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.587987+0000", 2024-09-15T02:49:06.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0230999999999999e-05 2024-09-15T02:49:06.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.87855124, 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016823131000000002, 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634136+0000", 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7940000000000001e-06 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634475+0000", 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033968099999999999 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634663+0000", 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018762999999999999 2024-09-15T02:49:06.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634681+0000", 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7974000000000001e-05 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634905+0000", 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022468600000000001 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634923+0000", 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7254000000000001e-05 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650923+0000", 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015999777999999999 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650957+0000", 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4334e-05 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 46/33 ECSubWrite(tid=32, reqid=client.4526.0:307, at_version=46'24, trim_to=0'0, roll_forward_to=44'23))", 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.881243088, 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082044130000000007, 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.534 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631442+0000", 2024-09-15T02:49:06.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.27e-07 2024-09-15T02:49:06.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631889+0000", 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044670300000000001 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631941+0000", 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1823999999999997e-05 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.631956+0000", 2024-09-15T02:49:06.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5143999999999999e-05 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632032+0000", 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6149999999999994e-05 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.632044+0000", 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2282e-05 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.639605+0000", 2024-09-15T02:49:06.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075610620000000003 2024-09-15T02:49:06.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:50.639646+0000", 2024-09-15T02:49:06.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0821000000000001e-05 2024-09-15T02:49:06.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.542 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.542 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 49/33 ECSubWrite(tid=155, reqid=client.4526.0:355, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:06.542 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:06.543 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.837477116000001, 2024-09-15T02:49:06.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019850428999999999, 2024-09-15T02:49:06.543 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.543 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.543 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9999999999999997e-07 2024-09-15T02:49:06.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675436+0000", 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227369 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675628+0000", 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019193300000000001 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675646+0000", 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7830999999999999e-05 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.676067+0000", 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042108899999999998 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.676096+0000", 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9149000000000002e-05 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695024+0000", 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018927979000000001 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695058+0000", 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4479e-05 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 49/33 ECSubWrite(tid=156, reqid=client.4526.0:356, at_version=49'19, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:06.548 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.829136599, 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011525719, 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683550+0000", 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.03e-06 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684004+0000", 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045311 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684167+0000", 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163869 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684188+0000", 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0190000000000002e-05 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684753+0000", 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056561700000000003 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684768+0000", 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4807000000000001e-05 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695067+0000", 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010299341 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695074+0000", 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7549999999999997e-06 2024-09-15T02:49:06.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 49/33 ECSubWrite(tid=178, reqid=client.4526.0:372, at_version=49'19, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.713790+0000", 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.798895124, 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087937559999999998, 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.713790+0000", 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.713790+0000", 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.713791+0000", 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0300000000000001e-06 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714120+0000", 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032886599999999998 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714161+0000", 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1090999999999997e-05 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714171+0000", 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0548e-05 2024-09-15T02:49:06.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714229+0000", 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7646999999999999e-05 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.714239+0000", 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0227999999999999e-05 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.722547+0000", 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083076820000000003 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.722584+0000", 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6664e-05 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:06.557 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.763094177999999, 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016121677000000001, 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749593+0000", 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.708e-06 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749692+0000", 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9852000000000006e-05 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749755+0000", 2024-09-15T02:49:06.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2292999999999995e-05 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749767+0000", 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2646000000000001e-05 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749863+0000", 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5872999999999995e-05 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749883+0000", 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0329000000000001e-05 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.765618+0000", 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015734416000000001 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.765712+0000", 2024-09-15T02:49:06.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4560000000000003e-05 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:395 3.4s0 3:2d69f33a:::smithi19125844-35:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.759884242, 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006832917, 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:06.562 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 395 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752802+0000", 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0869999999999999e-06 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752808+0000", 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6250000000000001e-06 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752844+0000", 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5979999999999998e-05 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752855+0000", 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0574000000000001e-05 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.754604+0000", 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017494609999999999 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.754741+0000", 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000136193 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759634+0000", 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048929969999999996 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:396 3.4s0 3:2d69f33a:::smithi19125844-35:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752913+0000", 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.759771841999999, 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008566957, 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.567 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 396 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752913+0000", 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752913+0000", 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752914+0000", 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752921+0000", 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8929999999999999e-06 2024-09-15T02:49:06.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752946+0000", 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4474e-05 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752954+0000", 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1880000000000002e-06 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.755131+0000", 2024-09-15T02:49:06.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002176978 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.755267+0000", 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135907 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761480+0000", 2024-09-15T02:49:06.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006213405 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.235891675, 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010710503999999999, 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276793+0000", 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.276794+0000", 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277315+0000", 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052080900000000001 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277480+0000", 2024-09-15T02:49:06.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016516400000000001 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277511+0000", 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0750999999999997e-05 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277623+0000", 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112213 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.277635+0000", 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1899000000000001e-05 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287471+0000", 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098365050000000006 2024-09-15T02:49:06.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287504+0000", 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2462999999999997e-05 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.222625652, 2024-09-15T02:49:06.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016610296, 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290061+0000", 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.327e-06 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290153+0000", 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2641999999999998e-05 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290291+0000", 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013733500000000001 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290300+0000", 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0459999999999994e-06 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290359+0000", 2024-09-15T02:49:06.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9695999999999999e-05 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290374+0000", 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4831000000000001e-05 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306642+0000", 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016267321000000001 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306670+0000", 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8098000000000001e-05 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.222512588000001, 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016516073999999999, 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.582 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:06.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290173+0000", 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5400000000000002e-07 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290241+0000", 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7681000000000005e-05 2024-09-15T02:49:06.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290326+0000", 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5348999999999999e-05 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290330+0000", 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.498e-06 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297821+0000", 2024-09-15T02:49:06.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074905040000000003 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297832+0000", 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0905e-05 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306676+0000", 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088443059999999997 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306688+0000", 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2377e-05 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=202, reqid=client.4526.0:432, at_version=50'61, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.296969+0000", 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.215715660000001, 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097351390000000003, 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.296969+0000", 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.296969+0000", 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.296970+0000", 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7699999999999992e-07 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297033+0000", 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2920000000000001e-05 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297081+0000", 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7655999999999999e-05 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297088+0000", 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.973e-06 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304585+0000", 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074971949999999999 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304596+0000", 2024-09-15T02:49:06.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306691+0000", 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020950539999999998 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306704+0000", 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3756e-05 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs5 50/33 ECSubWrite(tid=39, reqid=client.4526.0:423, at_version=0'0, trim_to=0'0, roll_forward_to=50'9))", 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.297010+0000", 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.215674579, 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067341830000000004, 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297010+0000", 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297010+0000", 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297012+0000", 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.184e-06 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297027+0000", 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5418e-05 2024-09-15T02:49:06.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297070+0000", 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2843000000000003e-05 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297082+0000", 2024-09-15T02:49:06.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-15T02:49:06.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297166+0000", 2024-09-15T02:49:06.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4474000000000005e-05 2024-09-15T02:49:06.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297179+0000", 2024-09-15T02:49:06.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3371e-05 2024-09-15T02:49:06.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303705+0000", 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065259749999999998 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303745+0000", 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9178000000000003e-05 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=203, reqid=client.4526.0:433, at_version=50'62, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.298710+0000", 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.213975206000001, 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080079529999999999, 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298710+0000", 2024-09-15T02:49:06.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298710+0000", 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298711+0000", 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5600000000000005e-07 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298732+0000", 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1503000000000001e-05 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298772+0000", 2024-09-15T02:49:06.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0191000000000002e-05 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298779+0000", 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7989999999999996e-06 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305762+0000", 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069827149999999996 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305771+0000", 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9090000000000006e-06 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306709+0000", 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093796299999999999 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306718+0000", 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1169999999999993e-06 2024-09-15T02:49:06.602 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=204, reqid=client.4526.0:438, at_version=50'64, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.300087+0000", 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.212597994999999, 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072160439999999996, 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300087+0000", 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300087+0000", 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300088+0000", 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5199999999999995e-07 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300115+0000", 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7169999999999999e-05 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300170+0000", 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4527999999999997e-05 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300176+0000", 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5880000000000003e-06 2024-09-15T02:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306001+0000", 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058244480000000003 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306011+0000", 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0394000000000001e-05 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.307269+0000", 2024-09-15T02:49:06.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012578120000000001 2024-09-15T02:49:06.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.307303+0000", 2024-09-15T02:49:06.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4251999999999997e-05 2024-09-15T02:49:06.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.608 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.609 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=61, reqid=client.4526.0:512, at_version=60'24, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:06.609 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.248578+0000", 2024-09-15T02:49:06.609 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.2641068120000001, 2024-09-15T02:49:06.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022275555999999998, 2024-09-15T02:49:06.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.609 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.248578+0000", 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.248578+0000", 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.248579+0000", 2024-09-15T02:49:06.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.06e-07 2024-09-15T02:49:06.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249504+0000", 2024-09-15T02:49:06.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000925583 2024-09-15T02:49:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249591+0000", 2024-09-15T02:49:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7118000000000002e-05 2024-09-15T02:49:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249614+0000", 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2520000000000001e-05 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249716+0000", 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010254 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249727+0000", 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0754000000000001e-05 2024-09-15T02:49:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270830+0000", 2024-09-15T02:49:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021102889 2024-09-15T02:49:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270854+0000", 2024-09-15T02:49:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3646e-05 2024-09-15T02:49:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=62, reqid=client.4526.0:513, at_version=60'25, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.256424+0000", 2024-09-15T02:49:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.2562614619999999, 2024-09-15T02:49:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014458907, 2024-09-15T02:49:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256424+0000", 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256424+0000", 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256425+0000", 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4139999999999999e-06 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256529+0000", 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010399999999999999 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256595+0000", 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5988000000000006e-05 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256607+0000", 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1613e-05 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.260325+0000", 2024-09-15T02:49:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003718341 2024-09-15T02:49:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.260341+0000", 2024-09-15T02:49:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6597000000000001e-05 2024-09-15T02:49:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270869+0000", 2024-09-15T02:49:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010527549000000001 2024-09-15T02:49:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270882+0000", 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3405e-05 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=63, reqid=client.4526.0:514, at_version=60'26, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.259676+0000", 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.2530093999999998, 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011625017, 2024-09-15T02:49:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259676+0000", 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259676+0000", 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259677+0000", 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259809+0000", 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013223600000000001 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259862+0000", 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2568999999999998e-05 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4239999999999993e-06 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.266670+0000", 2024-09-15T02:49:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068003999999999998 2024-09-15T02:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.266685+0000", 2024-09-15T02:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4861000000000001e-05 2024-09-15T02:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271278+0000", 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045931949999999996 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271301+0000", 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2104e-05 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=64, reqid=client.4526.0:515, at_version=60'27, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.262528+0000", 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.2501568559999998, 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087815819999999996, 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.262528+0000", 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.262528+0000", 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.262529+0000", 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3399999999999998e-07 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.262548+0000", 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8587999999999999e-05 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.262588+0000", 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0256000000000002e-05 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.262594+0000", 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1840000000000004e-06 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270176+0000", 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075821990000000004 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270188+0000", 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1623999999999999e-05 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:06.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271303+0000", 2024-09-15T02:49:06.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001115593 2024-09-15T02:49:06.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:06.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:06.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:06.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271310+0000", 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3040000000000001e-06 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:06.637 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:06.639 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:06.639 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:06.635+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:06.739 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:06.737+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:06.785 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:06.785 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:06.785 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:06.785 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:06.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:06.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:06.840 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:06.837+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:06.890 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:06.890 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:06.890 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:06.890 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:06.890 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:06.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:06.942 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:06.937+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:06.965 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:06.994 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:07.024 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:07.042 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:07.042 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:07.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:07.042 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:07.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.462783698999999, 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014940116999999999, 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577945+0000", 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.037e-06 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:49:07.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035186899999999999 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578459+0000", 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016206200000000001 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578482+0000", 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3070000000000001e-05 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578643+0000", 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016161999999999999 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578670+0000", 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592839+0000", 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014168557 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592884+0000", 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4786000000000001e-05 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.047 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.406521563999998, 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016865508000000001, 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0999999999999999e-07 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634501+0000", 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029489499999999999 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634605+0000", 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103488 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634624+0000", 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9530000000000001e-05 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634715+0000", 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1284000000000002e-05 2024-09-15T02:49:07.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634741+0000", 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5350999999999998e-05 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651036+0000", 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016295245999999999 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651071+0000", 2024-09-15T02:49:07.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5203999999999997e-05 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.290997828, 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085723550000000003, 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749729+0000", 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749730+0000", 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1500000000000004e-07 2024-09-15T02:49:07.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749937+0000", 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020695800000000001 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750030+0000", 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3487999999999997e-05 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750043+0000", 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2322e-05 2024-09-15T02:49:07.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750162+0000", 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119388 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.750182+0000", 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.02e-05 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758264+0000", 2024-09-15T02:49:07.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080818000000000001 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758302+0000", 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7583999999999998e-05 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=180, reqid=client.4526.0:385, at_version=49'53, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:07.056 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.281126232, 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010517344, 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:07.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759603+0000", 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7179999999999999e-06 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759826+0000", 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0002231 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759971+0000", 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014553100000000001 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760158+0000", 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018628199999999999 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760305+0000", 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014718500000000001 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760353+0000", 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8418999999999998e-05 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770084+0000", 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097306449999999996 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770118+0000", 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4464000000000001e-05 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.060 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.279846888, 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078090939999999999, 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760880+0000", 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760882+0000", 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.145e-06 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761368+0000", 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048694100000000001 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761447+0000", 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8586999999999998e-05 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761462+0000", 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4588e-05 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761530+0000", 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8718000000000002e-05 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761543+0000", 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.274e-05 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768640+0000", 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070968000000000003 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.768689+0000", 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9574999999999999e-05 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.065 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.768133197999999, 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090363309999999999, 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272594+0000", 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.34e-07 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.272948+0000", 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000353715 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273002+0000", 2024-09-15T02:49:07.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3458999999999998e-05 2024-09-15T02:49:07.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273014+0000", 2024-09-15T02:49:07.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1997000000000001e-05 2024-09-15T02:49:07.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273101+0000", 2024-09-15T02:49:07.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7424999999999997e-05 2024-09-15T02:49:07.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.273113+0000", 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1558e-05 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.281599+0000", 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084866360000000005 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.281630+0000", 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1207000000000001e-05 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:07.072 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.756287235, 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015418308, 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:07.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284441+0000", 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.251e-06 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284558+0000", 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011635 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284645+0000", 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6879000000000006e-05 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284653+0000", 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8100000000000004e-06 2024-09-15T02:49:07.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284760+0000", 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106585 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284774+0000", 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4260000000000001e-05 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299827+0000", 2024-09-15T02:49:07.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015052345 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299858+0000", 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1828000000000001e-05 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.077 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=62, reqid=client.4526.0:418, at_version=50'6, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.756164633000001, 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015317982000000001, 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:07.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284595+0000", 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2381e-05 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284673+0000", 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7855000000000006e-05 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284681+0000", 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9910000000000002e-06 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289497+0000", 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048157060000000003 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289509+0000", 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.166e-05 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299868+0000", 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010359454000000001 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299881+0000", 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2622e-05 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.082 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=63, reqid=client.4526.0:419, at_version=50'7, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.75121515, 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014955008000000001, 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289513+0000", 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.569999999999999e-07 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289540+0000", 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6984000000000001e-05 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289594+0000", 2024-09-15T02:49:07.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4117999999999999e-05 2024-09-15T02:49:07.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289608+0000", 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4219999999999999e-05 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299620+0000", 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010011978 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299633+0000", 2024-09-15T02:49:07.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2564e-05 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304436+0000", 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048034269999999999 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304467+0000", 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0759999999999997e-05 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.749764254, 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014581759999999999, 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290964+0000", 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6499999999999998e-07 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291188+0000", 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224513 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291233+0000", 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4665e-05 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291241+0000", 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8019999999999992e-06 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291323+0000", 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2284e-05 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291336+0000", 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2906e-05 2024-09-15T02:49:07.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305516+0000", 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014179916000000001 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305545+0000", 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8809000000000001e-05 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.094 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:07.094 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:07.094 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.749376154, 2024-09-15T02:49:07.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014221952, 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291352+0000", 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6000000000000004e-07 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291563+0000", 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000211387 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291616+0000", 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3294000000000002e-05 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291625+0000", 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8559999999999996e-06 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299094+0000", 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007469084 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299105+0000", 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0226e-05 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305552+0000", 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064471270000000004 2024-09-15T02:49:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305573+0000", 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1418000000000001e-05 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=202, reqid=client.4526.0:432, at_version=50'61, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.297119+0000", 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.743608201000001, 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084870129999999998, 2024-09-15T02:49:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297119+0000", 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297119+0000", 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297120+0000", 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.381e-06 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297159+0000", 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8161999999999998e-05 2024-09-15T02:49:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297207+0000", 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8541000000000002e-05 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297215+0000", 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7759999999999997e-06 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303660+0000", 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006445465 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303670+0000", 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8400000000000007e-06 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305587+0000", 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019168449999999999 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305606+0000", 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9003000000000001e-05 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=203, reqid=client.4526.0:433, at_version=50'62, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.298863+0000", 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.741864315000001, 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067607240000000001, 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298863+0000", 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298863+0000", 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298864+0000", 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1459999999999999e-06 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298891+0000", 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7118e-05 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298948+0000", 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7224e-05 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298959+0000", 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0706e-05 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304638+0000", 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056788819999999997 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304648+0000", 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7769999999999997e-06 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305608+0000", 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096046499999999997 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305624+0000", 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5406000000000001e-05 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=204, reqid=client.4526.0:438, at_version=50'64, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.300097+0000", 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.740630109, 2024-09-15T02:49:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063381089999999998, 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300097+0000", 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300097+0000", 2024-09-15T02:49:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300098+0000", 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4200000000000005e-07 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300121+0000", 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2884e-05 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300187+0000", 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6013000000000006e-05 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300194+0000", 2024-09-15T02:49:07.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0700000000000001e-06 2024-09-15T02:49:07.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304854+0000", 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004659959 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304864+0000", 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6430000000000007e-06 2024-09-15T02:49:07.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306408+0000", 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001544562 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306435+0000", 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7135999999999999e-05 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 60/33 ECSubWrite(tid=76, reqid=client.4526.0:530, at_version=60'53, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.271865+0000", 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.7688623159999999, 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081100259999999993, 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.117 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271865+0000", 2024-09-15T02:49:07.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271865+0000", 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271866+0000", 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.272364+0000", 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049823900000000004 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.272412+0000", 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7339999999999997e-05 2024-09-15T02:49:07.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.272421+0000", 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0969999999999995e-06 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.273529+0000", 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011077179999999999 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.273541+0000", 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2700999999999999e-05 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.279953+0000", 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064116199999999998 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.279975+0000", 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2050999999999999e-05 2024-09-15T02:49:07.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.6256486690000003, 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024987760000000001, 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7300000000000001e-07 2024-09-15T02:49:07.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420805+0000", 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057259099999999999 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420899+0000", 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4377999999999997e-05 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420915+0000", 2024-09-15T02:49:07.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5883e-05 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.421162+0000", 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024698199999999997 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.421192+0000", 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9342000000000001e-05 2024-09-15T02:49:07.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440023+0000", 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018831260999999998 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440066+0000", 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3531e-05 2024-09-15T02:49:07.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.128 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.128 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.128 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/47 ECSubWrite(tid=288, reqid=client.4526.0:542, at_version=60'32, trim_to=0'0, roll_forward_to=48'29))", 2024-09-15T02:49:07.128 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:07.129 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.6161285859999999, 2024-09-15T02:49:07.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011886391, 2024-09-15T02:49:07.129 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.129 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.129 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424601+0000", 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1330000000000002e-06 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424662+0000", 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1493000000000002e-05 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424768+0000", 2024-09-15T02:49:07.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010543 2024-09-15T02:49:07.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424786+0000", 2024-09-15T02:49:07.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.876e-05 2024-09-15T02:49:07.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424886+0000", 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9922999999999999e-05 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424906+0000", 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9110000000000002e-05 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436423+0000", 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011517135 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436485+0000", 2024-09-15T02:49:07.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2407000000000001e-05 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=54, reqid=client.4526.0:537, at_version=60'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.6152261399999999, 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014882912, 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.138 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425503+0000", 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425746+0000", 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024368799999999999 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425834+0000", 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7188999999999995e-05 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425848+0000", 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4107000000000001e-05 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429379+0000", 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035311700000000001 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429394+0000", 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5582000000000001e-05 2024-09-15T02:49:07.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440351+0000", 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010956702 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440384+0000", 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2873999999999997e-05 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.142 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/47 ECSubWrite(tid=289, reqid=client.4526.0:543, at_version=60'33, trim_to=0'0, roll_forward_to=48'29))", 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.6122023739999998, 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011573594, 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428526+0000", 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.482e-06 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428559+0000", 2024-09-15T02:49:07.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2526e-05 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428636+0000", 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.674e-05 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428655+0000", 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9664999999999999e-05 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429356+0000", 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070097400000000004 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429370+0000", 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3755e-05 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440078+0000", 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010708255999999999 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440098+0000", 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0196000000000001e-05 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=55, reqid=client.4526.0:538, at_version=60'8, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:07.147 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.433907+0000", 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.606819974, 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0066815240000000003, 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.433907+0000", 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.433907+0000", 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.433909+0000", 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.434102+0000", 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019295900000000001 2024-09-15T02:49:07.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.434164+0000", 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2457000000000002e-05 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.434175+0000", 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0807e-05 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436452+0000", 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022773419999999999 2024-09-15T02:49:07.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436462+0000", 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440551+0000", 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040885849999999996 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440589+0000", 2024-09-15T02:49:07.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8201000000000002e-05 2024-09-15T02:49:07.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.152 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:07.152 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:07.152 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:07.153 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:07.035+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:07.154 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:07.154 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:07.154 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:07.154 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:07.155 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:07.155 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:07.149+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:07.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:07.193 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:07.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:07.254 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:07.253+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:07.255 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:07.259 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.259 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:07.287 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:07.313 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:07.342 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:07.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:07.335+0000 7efd227fc700 0 -- 172.21.15.6:0/3513656466 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7efd2418e980 msgr2=0x7efd24190d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T02:49:07.345 INFO:tasks.rados.rados.0.smithi191.stdout:218: done (3 left) 2024-09-15T02:49:07.346 INFO:tasks.rados.rados.0.smithi191.stdout:219: done (2 left) 2024-09-15T02:49:07.347 INFO:tasks.rados.rados.0.smithi191.stdout:221: done (1 left) 2024-09-15T02:49:07.347 INFO:tasks.rados.rados.0.smithi191.stdout:222: done (0 left) 2024-09-15T02:49:07.347 INFO:tasks.rados.rados.0.smithi191.stdout:223: read oid 7 snap -1 2024-09-15T02:49:07.348 INFO:tasks.rados.rados.0.smithi191.stdout:223: expect (ObjNum 92 snap 15 seq_num 92) 2024-09-15T02:49:07.348 INFO:tasks.rados.rados.0.smithi191.stdout:224: read oid 11 snap 4 2024-09-15T02:49:07.348 INFO:tasks.rados.rados.0.smithi191.stdout:224: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:49:07.348 INFO:tasks.rados.rados.0.smithi191.stdout:225: snap_create 2024-09-15T02:49:07.355 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:07.347+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:07.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:49:07.365 INFO:tasks.rados.rados.0.smithi191.stdout:224: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:49:07.370 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.370 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.375 INFO:tasks.rados.rados.0.smithi191.stdout:223: expect (ObjNum 92 snap 15 seq_num 92) 2024-09-15T02:49:07.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:07.455 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:07.449+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:07.474 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.474 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:07.557 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:07.554+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:07.577 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:07.585 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.586 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:07.607 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:07.611 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:07.612 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:07.612 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:07.612 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:07.658 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:07.653+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:07.692 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.692 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:07.687+0000 7fbbd3fff700 0 --2- 172.21.15.6:0/3133099078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbd414a000 0x7fbbd414a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T02:49:07.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:07.759 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:07.753+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:07.798 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.798 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:07.860 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:07.853+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:07.915 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:07.915 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:07.925 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:07.961 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:07.961+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:08.021 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.021 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.029 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:08.062 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:08.055+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:08.129 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.129 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.138 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:08.163 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:08.159+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:08.241 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.241 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.250 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:49:08.265 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:08.259+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:08.335 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 10] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:49:08.335 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:49:08.336 INFO:tasks.thrashosds.thrasher:inject_pause on osd.15 2024-09-15T02:49:08.336 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T02:49:08.336 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T02:49:08.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok config set filestore_inject_stall 3 2024-09-15T02:49:08.347 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.347 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.360 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:49:08.361 INFO:tasks.rados.rados.0.smithi191.stdout:223: done (2 left) 2024-09-15T02:49:08.361 INFO:tasks.rados.rados.0.smithi191.stdout:224: done (1 left) 2024-09-15T02:49:08.361 INFO:tasks.rados.rados.0.smithi191.stdout:225: done (0 left) 2024-09-15T02:49:08.361 INFO:tasks.rados.rados.0.smithi191.stdout:226: rmattr oid 17 current snap is 17 2024-09-15T02:49:08.361 INFO:tasks.rados.rados.0.smithi191.stdout:226: done (0 left) 2024-09-15T02:49:08.362 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 11 current snap is 17 2024-09-15T02:49:08.362 INFO:tasks.rados.rados.0.smithi191.stdout:227: seq_num 100 ranges {1589248=524288,2113536=753664,2867200=409600} 2024-09-15T02:49:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:49:08.365 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:08.359+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:08.377 INFO:tasks.rados.rados.0.smithi191.stdout:227: writing smithi19125844-11 from 1589248 to 2113536 tid 1 2024-09-15T02:49:08.381 INFO:tasks.rados.rados.0.smithi191.stdout:227: writing smithi19125844-11 from 2113536 to 2867200 tid 2 2024-09-15T02:49:08.384 INFO:tasks.rados.rados.0.smithi191.stdout:227: writing smithi19125844-11 from 2867200 to 3276800 tid 3 2024-09-15T02:49:08.384 INFO:tasks.rados.rados.0.smithi191.stdout:228: read oid 20 snap -1 2024-09-15T02:49:08.384 INFO:tasks.rados.rados.0.smithi191.stdout:228: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:49:08.384 INFO:tasks.rados.rados.0.smithi191.stdout:229: snap_create 2024-09-15T02:49:08.393 INFO:tasks.rados.rados.0.smithi191.stdout:228: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:49:08.414 INFO:tasks.rados.rados.0.smithi191.stdout:227: finishing write tid 1 to smithi19125844-11 2024-09-15T02:49:08.415 INFO:tasks.rados.rados.0.smithi191.stdout:227: finishing write tid 2 to smithi19125844-11 2024-09-15T02:49:08.415 INFO:tasks.rados.rados.0.smithi191.stdout:227: finishing write tid 3 to smithi19125844-11 2024-09-15T02:49:08.448 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:08.449 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T02:49:08.449 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:08.461 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.461 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.466 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:08.462+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:08.471 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:08.567 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:08.565+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:08.614 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.614 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.624 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:08.652 INFO:tasks.rados.rados.0.smithi191.stdout:227: finishing write tid 4 to smithi19125844-11 2024-09-15T02:49:08.655 INFO:tasks.rados.rados.0.smithi191.stdout:227: finishing write tid 5 to smithi19125844-11 2024-09-15T02:49:08.656 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 11 v 22 (ObjNum 100 snap 17 seq_num 100) dirty exists 2024-09-15T02:49:08.656 INFO:tasks.rados.rados.0.smithi191.stdout:227: left oid 11 (ObjNum 100 snap 17 seq_num 100) 2024-09-15T02:49:08.668 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:08.666+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:08.722 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.722 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.731 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:08.772 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:08.767+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:08.870 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:08.863+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:08.876 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.876 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:08.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:08.971 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:08.964+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:08.992 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:08.992 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:09.072 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:09.070+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:09.148 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.148 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:09.173 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:09.173+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:09.251 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.251 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:09.274 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:09.269+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:09.362 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.362 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:227: done (2 left) 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:228: done (1 left) 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:229: done (0 left) 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:230: read oid 11 snap -1 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:230: expect (ObjNum 100 snap 17 seq_num 100) 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:231: copy_from oid 5 from oid 17 current snap is 18 2024-09-15T02:49:09.366 INFO:tasks.rados.rados.0.smithi191.stdout:232: setattr oid 32 current snap is 18 2024-09-15T02:49:09.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:09.373 INFO:tasks.rados.rados.0.smithi191.stdout:233: snap_remove snap 15 2024-09-15T02:49:09.375 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:09.367+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:09.466 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.466 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:09.476 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:09.468+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:09.577 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:09.568+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:09.578 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.578 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:09.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:09.678 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:09.674+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:09.690 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.690 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:09.779 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:09.774+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:09.790 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:09.800 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.800 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:09.821 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:09.850 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:09.879 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:09.879 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:09.874+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:09.906 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:09.908 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:09.908 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:09.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:09.934 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:09.964 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:09.981 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:09.978+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:09.991 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:10.013 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:10.013 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:10.016 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:10.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:10.043 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:10.071 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:10.081 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:10.078+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:10.097 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:10.120 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:10.120 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:10.124 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:10.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:10.152 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:10.183 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:10.176+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:10.225 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:10.225 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:10.234 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:10.284 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:10.276+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:10.343 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:10.344 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:10.344 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:10.344 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:10.353 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:233: done (3 left) 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:234: rollback oid 27 current snap is 18 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 27 to 14 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 32 v 53 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:231: finishing copy_from racing read to smithi19125844-5 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:231: finishing copy_from to smithi19125844-5 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 5 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T02:49:10.371 INFO:tasks.rados.rados.0.smithi191.stdout:230: expect (ObjNum 100 snap 17 seq_num 100) 2024-09-15T02:49:10.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:49:10.385 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:10.380+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:10.455 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:10.455 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:10.455 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:10.456 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:10.456 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:10.464 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:10.486 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:10.480+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:10.494 INFO:tasks.rados.rados.0.smithi191.stdout:230: done (3 left) 2024-09-15T02:49:10.494 INFO:tasks.rados.rados.0.smithi191.stdout:231: done (2 left) 2024-09-15T02:49:10.494 INFO:tasks.rados.rados.0.smithi191.stdout:232: done (1 left) 2024-09-15T02:49:10.494 INFO:tasks.rados.rados.0.smithi191.stdout:235: delete oid 29 current snap is 18 2024-09-15T02:49:10.495 INFO:tasks.rados.rados.0.smithi191.stdout:234: finishing rollback tid 0 to smithi19125844-27 2024-09-15T02:49:10.495 INFO:tasks.rados.rados.0.smithi191.stdout:234: finishing rollback tid 1 to smithi19125844-27 2024-09-15T02:49:10.495 INFO:tasks.rados.rados.0.smithi191.stdout:234: finishing rollback tid 2 to smithi19125844-27 2024-09-15T02:49:10.495 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 27 v 52 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:234: done (1 left) 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:235: done (0 left) 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:236: read oid 28 snap -1 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:236: expect (ObjNum 90 snap 14 seq_num 90) 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:237: read oid 41 snap -1 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:237: expect (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:238: read oid 42 snap -1 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:238: expect (ObjNum 96 snap 15 seq_num 96) 2024-09-15T02:49:10.501 INFO:tasks.rados.rados.0.smithi191.stdout:239: rollback oid 4 current snap is 18 2024-09-15T02:49:10.502 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 4 to 10 2024-09-15T02:49:10.502 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 30 current snap is 18 2024-09-15T02:49:10.502 INFO:tasks.rados.rados.0.smithi191.stdout:240: seq_num 102 ranges {491520=557056,1048576=638976,1687552=524288} 2024-09-15T02:49:10.508 INFO:tasks.rados.rados.0.smithi191.stdout:240: writing smithi19125844-30 from 491520 to 1048576 tid 1 2024-09-15T02:49:10.514 INFO:tasks.rados.rados.0.smithi191.stdout:240: writing smithi19125844-30 from 1048576 to 1687552 tid 2 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:240: writing smithi19125844-30 from 1687552 to 2211840 tid 3 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:241: read oid 45 snap -1 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:241: expect deleted 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:242: read oid 8 snap -1 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:242: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:243: delete oid 40 current snap is 18 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:239: finishing rollback tid 0 to smithi19125844-4 2024-09-15T02:49:10.519 INFO:tasks.rados.rados.0.smithi191.stdout:237: expect (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:49:10.519 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:10.549 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:10.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:10.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:10.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:10.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:10.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:871775f5:::smithi19125844-15:c, version: 48'19, data_included: [0~61440], data_size: 61440, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:871775f5:::smithi19125844-15:c@48'19, size: 245760, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:245760, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.574 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.127125114, 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80849678800000002, 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.575 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442896+0000", 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0449999999999999e-06 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442986+0000", 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9937000000000003e-05 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.576 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.246998+0000", 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80401223499999996 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247018+0000", 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9735999999999999e-05 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247164+0000", 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000146093 2024-09-15T02:49:10.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.251392+0000", 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042277419999999996 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:801a1da7:::smithi19125844-39:e, version: 49'21, data_included: [0~159744], data_size: 159744, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:e@49'21, size: 638976, copy_subset: [], clone_subset: {}, snapset: e=[]:{e=[e,b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:638976, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:10.578 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.304395059999999, 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010047222999999999, 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265626+0000", 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.325e-06 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265815+0000", 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018882999999999999 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.580 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265866+0000", 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0565000000000001e-05 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265876+0000", 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0273e-05 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.266085+0000", 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000209049 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.581 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.275672+0000", 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095871810000000002 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:10.582 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.285797096, 2024-09-15T02:49:10.584 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087482150000000002, 2024-09-15T02:49:10.584 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.585 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.586 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284497+0000", 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027328299999999998 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284538+0000", 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1449999999999998e-05 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.587 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284547+0000", 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1020000000000003e-06 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284639+0000", 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1615999999999998e-05 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.588 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.292971+0000", 2024-09-15T02:49:10.589 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083324509999999994 2024-09-15T02:49:10.589 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.589 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.589 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.589 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.589 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [262144~221184], data_size: 221184, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1933312, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.281481724000001, 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014791191, 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288540+0000", 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9369999999999998e-06 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.591 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288962+0000", 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042141100000000001 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289058+0000", 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6339e-05 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289089+0000", 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0867000000000001e-05 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.592 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289255+0000", 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016596199999999999 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.303330+0000", 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014074675 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.593 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:e, version: 49'26, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:e@49'26, size: 1081344, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.261017946000001, 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007188888, 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.594 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309004+0000", 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4529999999999999e-06 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309408+0000", 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000404798 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.595 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309488+0000", 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9389999999999997e-05 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309520+0000", 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2509999999999999e-05 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.596 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309641+0000", 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120958 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.316191+0000", 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0065497790000000004 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.597 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.244918070000001, 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012696989000000001, 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.598 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325103+0000", 2024-09-15T02:49:10.599 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.333e-06 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325526+0000", 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000422788 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325605+0000", 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8932999999999998e-05 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.600 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325624+0000", 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8525999999999998e-05 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325780+0000", 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015595999999999999 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.337799+0000", 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012019449 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.601 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.22755298, 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01280455, 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.602 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342469+0000", 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6929999999999999e-06 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.603 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342850+0000", 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000381462 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342951+0000", 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000100559 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342971+0000", 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9720000000000001e-05 2024-09-15T02:49:10.604 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.343066+0000", 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5088999999999997e-05 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.355272+0000", 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012206027 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.605 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [524288~81920], data_size: 81920, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2424832, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.359417+0000", 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.210602817, 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045017099999999999, 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359417+0000", 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.606 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359417+0000", 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359419+0000", 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7099999999999999e-06 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359576+0000", 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000156849 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.607 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359660+0000", 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3772999999999999e-05 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359688+0000", 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7813e-05 2024-09-15T02:49:10.608 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.359839+0000", 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015179200000000001 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.363919+0000", 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0040797730000000001 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.609 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.19985889, 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012408221000000001, 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370163+0000", 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.611 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370553+0000", 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038988100000000002 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370636+0000", 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2825999999999996e-05 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370655+0000", 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.916e-05 2024-09-15T02:49:10.612 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370778+0000", 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012348700000000001 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.382570+0000", 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01179122 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.613 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.183743107, 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070382149999999996, 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.614 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386278+0000", 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2699999999999999e-06 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386706+0000", 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042724800000000003 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.615 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386783+0000", 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6976000000000003e-05 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386809+0000", 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6143e-05 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386892+0000", 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3196999999999996e-05 2024-09-15T02:49:10.616 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.393315+0000", 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0064233809999999997 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.172766835999999, 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0076400929999999997, 2024-09-15T02:49:10.617 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.618 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397255+0000", 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.576e-06 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397632+0000", 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000377265 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397715+0000", 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3213000000000004e-05 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.619 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397743+0000", 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7229999999999998e-05 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397885+0000", 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014198699999999999 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.620 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.404893+0000", 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070088219999999996 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [786432~167936], data_size: 167936, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3817472, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.161004326, 2024-09-15T02:49:10.621 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053001360000000004, 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.622 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409017+0000", 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.406e-06 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409301+0000", 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028334600000000002 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409374+0000", 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3652999999999994e-05 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.623 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409421+0000", 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6879000000000003e-05 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409504+0000", 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2559999999999996e-05 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.414316+0000", 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048122920000000001 2024-09-15T02:49:10.624 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/53 ECSubWrite(tid=67, reqid=client.4526.0:522, at_version=60'31, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.3024724719999998, 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43332960599999998, 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.625 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8200000000000001e-07 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.626 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267569+0000", 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0713e-05 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699630+0000", 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43206165099999999 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699644+0000", 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3274e-05 2024-09-15T02:49:10.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699794+0000", 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000150719 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699816+0000", 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1895999999999998e-05 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700821+0000", 2024-09-15T02:49:10.628 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010047210000000001 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700877+0000", 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6249999999999998e-05 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 61/53 ECSubWrite(tid=70, reqid=osd.0.0:70, at_version=61'33, trim_to=0'0, roll_forward_to=60'31))", 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.334705+0000", 2024-09-15T02:49:10.629 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2353155759999996, 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003573559, 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334705+0000", 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334705+0000", 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334705+0000", 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9700000000000003e-07 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334739+0000", 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3785999999999998e-05 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334966+0000", 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00022745900000000001 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334976+0000", 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9850000000000001e-06 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335080+0000", 2024-09-15T02:49:10.632 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000103948 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335091+0000", 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1156e-05 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.338233+0000", 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0031421650000000001 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.633 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.338278+0000", 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4762999999999998e-05 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 61/53 ECSubWrite(tid=71, reqid=osd.0.0:71, at_version=61'35, trim_to=0'0, roll_forward_to=60'31))", 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.334742+0000", 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2352779890000001, 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0039809659999999998, 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.634 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334742+0000", 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334742+0000", 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334743+0000", 2024-09-15T02:49:10.635 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334773+0000", 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0477e-05 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335004+0000", 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00023140500000000001 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.636 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335023+0000", 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8385000000000001e-05 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.336685+0000", 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001662618 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.336696+0000", 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0737000000000001e-05 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.338694+0000", 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0019975739999999998 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.338723+0000", 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9379000000000001e-05 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.638 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 61/53 ECSubWrite(tid=99, reqid=osd.14.0:99, at_version=61'43, trim_to=0'0, roll_forward_to=59'41))", 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.334832+0000", 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2351880729999998, 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003464296, 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334832+0000", 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.639 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334832+0000", 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334832+0000", 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3799999999999999e-07 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.334861+0000", 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8186000000000001e-05 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.640 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335405+0000", 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00054432600000000003 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335409+0000", 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4830000000000001e-06 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335455+0000", 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6034e-05 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.641 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.335465+0000", 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2439999999999999e-06 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.338274+0000", 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0028092410000000001 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.338296+0000", 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2544e-05 2024-09-15T02:49:10.642 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.6s6 61/53 ECSubWrite(tid=295, reqid=osd.13.0:295, at_version=61'20, trim_to=0'0, roll_forward_to=60'16))", 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.342076+0000", 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2279439009999997, 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045016489999999999, 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.643 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342076+0000", 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342076+0000", 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342077+0000", 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.094e-06 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.644 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342103+0000", 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5193000000000001e-05 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342148+0000", 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5868000000000002e-05 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342156+0000", 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8259999999999993e-06 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.645 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342206+0000", 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9616000000000001e-05 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.342218+0000", 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2432999999999999e-05 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.346541+0000", 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043230439999999998 2024-09-15T02:49:10.646 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.346578+0000", 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6575000000000001e-05 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 61/53 ECSubWrite(tid=299, reqid=osd.13.0:292, at_version=0'0, trim_to=0'0, roll_forward_to=61'45))", 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.348071+0000", 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2219487920000001, 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036168099999999998, 2024-09-15T02:49:10.647 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348071+0000", 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348071+0000", 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.648 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348072+0000", 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000004e-07 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348084+0000", 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1986e-05 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348116+0000", 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1418000000000003e-05 2024-09-15T02:49:10.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348121+0000", 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4419999999999997e-06 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348170+0000", 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8628999999999998e-05 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.348178+0000", 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4109999999999996e-06 2024-09-15T02:49:10.650 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.351653+0000", 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0034744799999999998 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.351688+0000", 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5549999999999997e-05 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.651 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 61/53 ECSubWrite(tid=102, reqid=osd.14.0:99, at_version=0'0, trim_to=0'0, roll_forward_to=61'43))", 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2207234600000003, 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0054209050000000002, 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.652 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7499999999999999e-07 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349311+0000", 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3689999999999999e-05 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.653 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349385+0000", 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3937999999999996e-05 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349392+0000", 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0790000000000003e-06 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349455+0000", 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2742000000000003e-05 2024-09-15T02:49:10.654 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349465+0000", 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0259e-05 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354685+0000", 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052197889999999999 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.655 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354718+0000", 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2632999999999997e-05 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 61/53 ECSubWrite(tid=75, reqid=osd.0.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=61'35))", 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.350362+0000", 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2196585300000002, 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043902450000000001, 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:10.656 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350362+0000", 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350362+0000", 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350362+0000", 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2399999999999997e-07 2024-09-15T02:49:10.657 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350383+0000", 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0755e-05 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350420+0000", 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7036999999999998e-05 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350427+0000", 2024-09-15T02:49:10.658 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8859999999999998e-06 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350486+0000", 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8789000000000002e-05 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.350497+0000", 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1218e-05 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.659 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354723+0000", 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042257199999999997 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354752+0000", 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9016000000000001e-05 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:10.660 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:239: finishing rollback tid 1 to smithi19125844-4 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:239: finishing rollback tid 2 to smithi19125844-4 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 4 v 82 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:242: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:240: finishing write tid 1 to smithi19125844-30 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:236: expect (ObjNum 90 snap 14 seq_num 90) 2024-09-15T02:49:10.662 INFO:tasks.rados.rados.0.smithi191.stdout:238: expect (ObjNum 96 snap 15 seq_num 96) 2024-09-15T02:49:10.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:10.663 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:10.660+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:10.728 INFO:tasks.rados.rados.0.smithi191.stdout:240: finishing write tid 2 to smithi19125844-30 2024-09-15T02:49:10.728 INFO:tasks.rados.rados.0.smithi191.stdout:240: finishing write tid 3 to smithi19125844-30 2024-09-15T02:49:10.728 INFO:tasks.rados.rados.0.smithi191.stdout:240: finishing write tid 4 to smithi19125844-30 2024-09-15T02:49:10.728 INFO:tasks.rados.rados.0.smithi191.stdout:240: finishing write tid 5 to smithi19125844-30 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 30 v 57 (ObjNum 102 snap 18 seq_num 102) dirty exists 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:240: left oid 30 (ObjNum 102 snap 18 seq_num 102) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:236: done (7 left) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:237: done (6 left) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:238: done (5 left) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:239: done (4 left) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:240: done (3 left) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:241: done (2 left) 2024-09-15T02:49:10.729 INFO:tasks.rados.rados.0.smithi191.stdout:242: done (1 left) 2024-09-15T02:49:10.730 INFO:tasks.rados.rados.0.smithi191.stdout:243: done (0 left) 2024-09-15T02:49:10.730 INFO:tasks.rados.rados.0.smithi191.stdout:244: rollback oid 28 current snap is 18 2024-09-15T02:49:10.730 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 28 to 10 2024-09-15T02:49:10.730 INFO:tasks.rados.rados.0.smithi191.stdout:245: read oid 44 snap -1 2024-09-15T02:49:10.730 INFO:tasks.rados.rados.0.smithi191.stdout:245: expect (ObjNum 98 snap 15 seq_num 98) 2024-09-15T02:49:10.730 INFO:tasks.rados.rados.0.smithi191.stdout:246: snap_remove snap 12 2024-09-15T02:49:10.763 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:10.756+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:10.769 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:10.769 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:10.769 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:10.769 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:10.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:10.863 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:10.860+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:10.873 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:10.874 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:10.874 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:10.874 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:10.874 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:10.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:10.964 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:10.962+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 20.332776908, 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01495401, 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649418+0000", 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.553e-06 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649682+0000", 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026439200000000001 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649749+0000", 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6585e-05 2024-09-15T02:49:10.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649761+0000", 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1584e-05 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649838+0000", 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7866000000000003e-05 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649852+0000", 2024-09-15T02:49:10.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4e-05 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664337+0000", 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014485031000000001 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664370+0000", 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2999e-05 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:10.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.233898333999999, 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014289939, 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748295+0000", 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6199999999999999e-07 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748880+0000", 2024-09-15T02:49:10.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000584813 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748952+0000", 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1798000000000006e-05 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.748962+0000", 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0567999999999999e-05 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.749080+0000", 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117314 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.749096+0000", 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6282000000000001e-05 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.762525+0000", 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013428993 2024-09-15T02:49:10.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.762585+0000", 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9908999999999998e-05 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.222966275000001, 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015371035, 2024-09-15T02:49:10.996 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759228+0000", 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3599999999999999e-06 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759805+0000", 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057638899999999996 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759906+0000", 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010075499999999999 2024-09-15T02:49:10.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759920+0000", 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4166e-05 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.760017+0000", 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7647999999999997e-05 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:10.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.760038+0000", 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0290999999999999e-05 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.774535+0000", 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014497252 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.774598+0000", 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3174000000000002e-05 2024-09-15T02:49:11.000 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:422 3.cs0 3:319e3fcc:::smithi19125844-50:head [rollback a] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.715723795999999, 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028855308, 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.001 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 422 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.85e-07 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266475+0000", 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.527e-06 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266498+0000", 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3652999999999999e-05 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266521+0000", 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3006999999999998e-05 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268383+0000", 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018616730000000001 2024-09-15T02:49:11.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268454+0000", 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0479000000000001e-05 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275682+0000", 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072287380000000002 2024-09-15T02:49:11.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283977+0000", 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082944760000000003 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295223+0000", 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011245890999999999 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295325+0000", 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102079 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:423 3.cs0 3:319e3fcc:::smithi19125844-50:head [append 0~0] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.715653839000002, 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028955272000000001, 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.007 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 423 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266541+0000", 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5719999999999999e-06 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266548+0000", 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4899999999999997e-06 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266559+0000", 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1158000000000001e-05 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266564+0000", 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1630000000000004e-06 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276125+0000", 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095610759999999999 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276243+0000", 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117991 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276700+0000", 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00045704300000000002 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286454+0000", 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097542040000000007 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295420+0000", 2024-09-15T02:49:11.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089656649999999994 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295495+0000", 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4909999999999999e-05 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.707093462, 2024-09-15T02:49:11.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014524752, 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275101+0000", 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6899999999999996e-07 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275587+0000", 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000485976 2024-09-15T02:49:11.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275641+0000", 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4728999999999997e-05 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275650+0000", 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4749999999999993e-06 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275756+0000", 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105877 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275768+0000", 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2734999999999999e-05 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289496+0000", 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013727892 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289625+0000", 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012829900000000001 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.018 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:512 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~786432] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7388268460000003, 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026925933999999999, 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 512 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243369+0000", 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7709999999999998e-06 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.245943+0000", 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025742260000000002 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.245980+0000", 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6510000000000001e-05 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246008+0000", 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8286e-05 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246085+0000", 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6760000000000004e-05 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246181+0000", 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5546999999999998e-05 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.247292+0000", 2024-09-15T02:49:11.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011118090000000001 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261781+0000", 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014488861 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270200+0000", 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084192450000000005 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270292+0000", 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1918999999999999e-05 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:513 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~491520] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7359071530000003, 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024200269, 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.024 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.025 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 513 2024-09-15T02:49:11.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.025 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:11.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246289+0000", 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7329999999999999e-06 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248021+0000", 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017319410000000001 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248044+0000", 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3451999999999998e-05 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248057+0000", 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2745000000000001e-05 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.253501+0000", 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054438999999999998 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.253636+0000", 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000134587 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.254328+0000", 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000692474 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074557210000000002 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270429+0000", 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0086448540000000004 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270486+0000", 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7862000000000003e-05 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:514 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~278528] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7340658610000004, 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023758102, 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 514 2024-09-15T02:49:11.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248129+0000", 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4389999999999999e-06 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248897+0000", 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000768591 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248919+0000", 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1985999999999999e-05 2024-09-15T02:49:11.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248931+0000", 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1516e-05 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.257681+0000", 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087502150000000004 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.257788+0000", 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106774 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.258392+0000", 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060445900000000001 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266310+0000", 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079177050000000006 2024-09-15T02:49:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271830+0000", 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055203140000000001 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271886+0000", 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5102999999999997e-05 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:515 3.fs0 3:f0e056ea:::smithi19125844-7:head [setxattr _header (56) in=7b] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7332529379999997, 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023210873999999999, 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 515 2024-09-15T02:49:11.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248941+0000", 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.035e-06 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248945+0000", 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0330000000000002e-06 2024-09-15T02:49:11.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248957+0000", 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1669e-05 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248962+0000", 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2650000000000004e-06 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.260634+0000", 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011671488000000001 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.260737+0000", 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102816 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261347+0000", 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000610043 2024-09-15T02:49:11.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266315+0000", 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049678099999999996 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272042+0000", 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005727869 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272151+0000", 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010884599999999999 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:516 3.fs0 3:f0e056ea:::smithi19125844-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:11.045 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7332253099999999, 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.025266506000000001, 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 516 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:11.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248969+0000", 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4700000000000001e-07 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248971+0000", 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.588e-06 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248983+0000", 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1579999999999999e-05 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248995+0000", 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.185e-05 2024-09-15T02:49:11.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261706+0000", 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012710971 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261774+0000", 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8342000000000004e-05 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272225+0000", 2024-09-15T02:49:11.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010450971999999999 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272297+0000", 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1903999999999995e-05 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.274235+0000", 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019377520000000001 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:520 3.fs0 3:f39d968a:::smithi19125844-6:head [append 0~0] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7331934430000002, 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024220364000000001, 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.051 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 520 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2100000000000002e-07 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249003+0000", 2024-09-15T02:49:11.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4779999999999998e-06 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249010+0000", 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1799999999999999e-06 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249018+0000", 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1300000000000001e-06 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261843+0000", 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012824894999999999 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262019+0000", 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017557599999999999 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263083+0000", 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001064206 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266322+0000", 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032392810000000001 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273128+0000", 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068061969999999999 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273220+0000", 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2e-05 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:521 3.fs0 3:f39d968a:::smithi19125844-6:head [rollback 15] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.733169513, 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024414946999999999, 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 521 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.057 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:11.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3700000000000001e-07 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249027+0000", 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7819999999999998e-06 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249038+0000", 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0569e-05 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249043+0000", 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2129999999999999e-06 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263801+0000", 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014758603 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263989+0000", 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000188004 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.264897+0000", 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090739499999999997 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.268454+0000", 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035574119999999998 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273370+0000", 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049154689999999996 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273439+0000", 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9062999999999999e-05 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:522 3.fs0 3:f39d968a:::smithi19125844-6:head [append 0~0] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7331392020000003, 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45227964500000001, 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.062 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 522 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249055+0000", 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0800000000000005e-07 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249058+0000", 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.914e-06 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249068+0000", 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0883000000000001e-05 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249073+0000", 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5399999999999997e-06 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.265365+0000", 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016292469 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.265485+0000", 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011935 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266071+0000", 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058648000000000001 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271231+0000", 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051601140000000004 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.701218+0000", 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.42998620999999998 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.701334+0000", 2024-09-15T02:49:11.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116177 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 60/33 ECSubWrite(tid=75, reqid=client.4526.0:529, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.720731024, 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022861161000000001, 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261463+0000", 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.13e-06 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261848+0000", 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038425800000000001 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261906+0000", 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7896000000000003e-05 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261913+0000", 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8269999999999998e-06 2024-09-15T02:49:11.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261989+0000", 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.559e-05 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262002+0000", 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3291e-05 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284285+0000", 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022282659999999999 2024-09-15T02:49:11.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284323+0000", 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8509e-05 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 60/33 ECSubWrite(tid=76, reqid=client.4526.0:530, at_version=60'53, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.7117207670000001, 2024-09-15T02:49:11.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013872888, 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270473+0000", 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9599999999999999e-07 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271037+0000", 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00056329999999999998 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271079+0000", 2024-09-15T02:49:11.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2446e-05 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271086+0000", 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0439999999999998e-06 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276187+0000", 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051012849999999997 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276202+0000", 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4887e-05 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284331+0000", 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081288690000000004 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284345+0000", 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4361e-05 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 62/33 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=62'65, trim_to=0'0, roll_forward_to=61'63))", 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:06.347459+0000", 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.6347347069999998, 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013037906, 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347459+0000", 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347459+0000", 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347459+0000", 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9400000000000005e-07 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347508+0000", 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9029000000000001e-05 2024-09-15T02:49:11.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347591+0000", 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2656000000000003e-05 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347605+0000", 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4358e-05 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347729+0000", 2024-09-15T02:49:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012390200000000001 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347755+0000", 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6050999999999998e-05 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.360443+0000", 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012688084000000001 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.360497+0000", 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3232000000000001e-05 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=64, reqid=client.4526.0:610, at_version=66'54, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.510547+0000", 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.47164646500000001, 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01556914, 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.081 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510547+0000", 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510547+0000", 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510548+0000", 2024-09-15T02:49:11.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.643e-06 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511151+0000", 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060209599999999999 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511226+0000", 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5841999999999997e-05 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511238+0000", 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1731e-05 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511301+0000", 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2636000000000001e-05 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511316+0000", 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.472e-05 2024-09-15T02:49:11.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.525996+0000", 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014680901 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526116+0000", 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119571 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=65, reqid=client.4526.0:611, at_version=66'55, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.518894+0000", 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.46329962800000002, 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019660654, 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518894+0000", 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518894+0000", 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518895+0000", 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8299999999999995e-07 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519184+0000", 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00028936100000000002 2024-09-15T02:49:11.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519222+0000", 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7929999999999998e-05 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519231+0000", 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0399999999999998e-06 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519369+0000", 2024-09-15T02:49:11.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013751400000000001 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519435+0000", 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6125999999999997e-05 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.538522+0000", 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019087197 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.538554+0000", 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2502999999999997e-05 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=66, reqid=client.4526.0:612, at_version=66'56, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.526147+0000", 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.45604623599999999, 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012547018, 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526147+0000", 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526147+0000", 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526148+0000", 2024-09-15T02:49:11.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000003e-07 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526365+0000", 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000217116 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526422+0000", 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7333999999999997e-05 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526430+0000", 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1810000000000004e-06 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526815+0000", 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038553600000000002 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526830+0000", 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4423000000000001e-05 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.538666+0000", 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011836619 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.538694+0000", 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7894000000000001e-05 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:11.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:11.097 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:11.092+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:11.197 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:11.192+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:11.207 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:11.207 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:11.208 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:11.208 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:11.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:11.299 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:11.291+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:11.323 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:11.324 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:11.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:11.389 INFO:tasks.rados.rados.0.smithi191.stdout:246: done (2 left) 2024-09-15T02:49:11.390 INFO:tasks.rados.rados.0.smithi191.stdout:247: snap_remove snap 16 2024-09-15T02:49:11.400 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:11.398+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:49:11.486 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:11.486 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:11.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 38/33 ECSubWrite(tid=19, reqid=osd.15.0:19, at_version=38'14, trim_to=0'0, roll_forward_to=37'12))", 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:42.534265+0000", 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.947468468, 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010812166999999999, 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534265+0000", 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534265+0000", 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534266+0000", 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534315+0000", 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8541000000000002e-05 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534422+0000", 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107373 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.534438+0000", 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5906e-05 2024-09-15T02:49:11.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.536647+0000", 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002209262 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.536661+0000", 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4192000000000001e-05 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.545032+0000", 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083703709999999997 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:42.545078+0000", 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5948000000000001e-05 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:11.491 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.913068497000001, 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016202787999999999, 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568666+0000", 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0699999999999997e-07 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569158+0000", 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000491633 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569218+0000", 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0742999999999998e-05 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569231+0000", 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2455e-05 2024-09-15T02:49:11.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569315+0000", 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3701000000000004e-05 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569329+0000", 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4606e-05 2024-09-15T02:49:11.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.584839+0000", 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015510222000000001 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.584868+0000", 2024-09-15T02:49:11.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8821000000000001e-05 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 20.832119937999998, 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014988552, 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649615+0000", 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8800000000000002e-07 2024-09-15T02:49:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650002+0000", 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038781599999999997 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650043+0000", 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0312999999999998e-05 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650051+0000", 2024-09-15T02:49:11.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2390000000000002e-06 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650118+0000", 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7015999999999997e-05 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650127+0000", 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6200000000000005e-06 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664560+0000", 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014433692 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664603+0000", 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2268000000000003e-05 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.501 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.789811186000001, 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013640826, 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691924+0000", 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.544e-06 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692182+0000", 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025800999999999998 2024-09-15T02:49:11.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692284+0000", 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010141699999999999 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692295+0000", 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1447999999999999e-05 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692382+0000", 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7294999999999997e-05 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692401+0000", 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8635e-05 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.705531+0000", 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013129403 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.705564+0000", 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3074000000000002e-05 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=160, reqid=client.4526.0:364, at_version=49'23, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.784519912, 2024-09-15T02:49:11.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011544153, 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697215+0000", 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.046e-06 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697510+0000", 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00029446300000000002 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697550+0000", 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0247999999999998e-05 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697562+0000", 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2068e-05 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.699512+0000", 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019506219999999999 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.699531+0000", 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8343999999999999e-05 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708741+0000", 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092103159999999996 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708758+0000", 2024-09-15T02:49:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7045999999999999e-05 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.228160990999999, 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031206293, 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.511 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.512 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254463+0000", 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088986800000000004 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254519+0000", 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5640000000000003e-05 2024-09-15T02:49:11.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254529+0000", 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0152999999999999e-05 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254650+0000", 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012129300000000001 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254679+0000", 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9107000000000001e-05 2024-09-15T02:49:11.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284735+0000", 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.030055693000000001 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284779+0000", 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4199000000000002e-05 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.221267756, 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024332761000000001, 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260468+0000", 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.508e-06 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260940+0000", 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047216899999999998 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.261103+0000", 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016333 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.261116+0000", 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.327e-05 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.269681+0000", 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0085646730000000001 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.269691+0000", 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4339999999999998e-06 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284787+0000", 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015096058000000001 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284799+0000", 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2319e-05 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.213685534, 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01876924, 2024-09-15T02:49:11.520 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268049+0000", 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268412+0000", 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000363111 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268461+0000", 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9150000000000002e-05 2024-09-15T02:49:11.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268470+0000", 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5909999999999996e-06 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268578+0000", 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107676 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268591+0000", 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3733e-05 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286797+0000", 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.018205193000000001 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286818+0000", 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1012000000000001e-05 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.524 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=195, reqid=client.4526.0:411, at_version=50'34, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.211616958, 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014722531000000001, 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270118+0000", 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.012e-06 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270182+0000", 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4114000000000003e-05 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270229+0000", 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6473e-05 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270241+0000", 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2092999999999999e-05 2024-09-15T02:49:11.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275025+0000", 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047846529999999998 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275036+0000", 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0637e-05 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284820+0000", 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097838739999999997 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284840+0000", 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9675000000000001e-05 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:11.529 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.206321931000002, 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014093722, 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:11.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275413+0000", 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1459999999999999e-06 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277178+0000", 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017643979999999999 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277259+0000", 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1926000000000001e-05 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277268+0000", 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4249999999999998e-06 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277340+0000", 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2126999999999996e-05 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277360+0000", 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9678e-05 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289453+0000", 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012093051 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289506+0000", 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2970999999999998e-05 2024-09-15T02:49:11.533 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=196, reqid=client.4526.0:412, at_version=50'35, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275559+0000", 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.206174588, 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011112530000000001, 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275559+0000", 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275559+0000", 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275588+0000", 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8394999999999998e-05 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275613+0000", 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.563e-05 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275663+0000", 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9629999999999997e-05 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275671+0000", 2024-09-15T02:49:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2160000000000006e-06 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283502+0000", 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078303230000000001 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283518+0000", 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6617999999999999e-05 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286646+0000", 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031278539999999998 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286672+0000", 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5863999999999998e-05 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.277911+0000", 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.203822993999999, 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011518654999999999, 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277911+0000", 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277911+0000", 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277912+0000", 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.066e-06 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278153+0000", 2024-09-15T02:49:11.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024120700000000001 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278215+0000", 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1956000000000001e-05 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278225+0000", 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0292000000000001e-05 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278314+0000", 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8991000000000002e-05 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.278325+0000", 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0584000000000001e-05 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289387+0000", 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011061903 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289430+0000", 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2656e-05 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as1 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.169607821, 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012325150999999999, 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.543 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312128+0000", 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.59e-06 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312511+0000", 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038326000000000001 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312619+0000", 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010794 2024-09-15T02:49:11.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312637+0000", 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8104000000000001e-05 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312798+0000", 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016069599999999999 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312831+0000", 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3179999999999997e-05 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324419+0000", 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011588022999999999 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324451+0000", 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2357999999999997e-05 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.547 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 60/33 ECSubWrite(tid=75, reqid=client.4526.0:529, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.2198256010000001, 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014182907999999999, 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:11.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261909+0000", 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262920+0000", 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001011236 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262962+0000", 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1980999999999997e-05 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262988+0000", 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6571e-05 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263089+0000", 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100937 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263102+0000", 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3183999999999999e-05 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276063+0000", 2024-09-15T02:49:11.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012960833 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276091+0000", 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8002999999999999e-05 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 61/53 ECSubWrite(tid=99, reqid=osd.14.0:99, at_version=61'43, trim_to=0'0, roll_forward_to=59'41))", 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.334519+0000", 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.1472147389999998, 2024-09-15T02:49:11.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014171448, 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334519+0000", 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334519+0000", 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334520+0000", 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2599999999999998e-07 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334563+0000", 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3136999999999997e-05 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334622+0000", 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9162e-05 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334629+0000", 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0380000000000002e-06 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334682+0000", 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3012e-05 2024-09-15T02:49:11.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334690+0000", 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5290000000000008e-06 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.348649+0000", 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013958834 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.348691+0000", 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1409999999999998e-05 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 61/33 ECSubWrite(tid=52, reqid=osd.15.0:52, at_version=61'40, trim_to=0'0, roll_forward_to=59'36))", 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.334691+0000", 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.147042645, 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016138692, 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334691+0000", 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334691+0000", 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334692+0000", 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:49:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334707+0000", 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5034e-05 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334791+0000", 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4495999999999999e-05 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334796+0000", 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7829999999999998e-06 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.337673+0000", 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028774109999999999 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.337684+0000", 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0366e-05 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.350807+0000", 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013123374 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.350830+0000", 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2887999999999998e-05 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.561 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 62/33 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=62'48, trim_to=0'0, roll_forward_to=61'44))", 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:06.347594+0000", 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.1341398109999998, 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012425673999999999, 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347594+0000", 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347594+0000", 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347595+0000", 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5799999999999999e-07 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347620+0000", 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5384e-05 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347751+0000", 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013113600000000001 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347759+0000", 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6620000000000005e-06 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348433+0000", 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00067455 2024-09-15T02:49:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348451+0000", 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7963999999999999e-05 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.359959+0000", 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011507969999999999 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.360020+0000", 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0449999999999999e-05 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 62/33 ECSubWrite(tid=203, reqid=osd.11.0:203, at_version=62'33, trim_to=0'0, roll_forward_to=61'31))", 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:06.348777+0000", 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.1329569499999996, 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011198992, 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.566 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348777+0000", 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348777+0000", 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348777+0000", 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5400000000000002e-07 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348823+0000", 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5775000000000001e-05 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348903+0000", 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9812000000000001e-05 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348913+0000", 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0519999999999999e-05 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.349042+0000", 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000128632 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.349058+0000", 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6178e-05 2024-09-15T02:49:11.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.359941+0000", 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010883158 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.359976+0000", 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4563000000000001e-05 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 66/33 ECSubWrite(tid=64, reqid=client.4526.0:610, at_version=66'54, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.510782+0000", 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.97095221499999995, 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015282408000000001, 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510782+0000", 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510782+0000", 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510783+0000", 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.567e-06 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512149+0000", 2024-09-15T02:49:11.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001365502 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512202+0000", 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3256e-05 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512213+0000", 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.075e-05 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512281+0000", 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7787999999999995e-05 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512299+0000", 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8754999999999999e-05 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.525981+0000", 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013681759999999999 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526064+0000", 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3029999999999996e-05 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 66/33 ECSubWrite(tid=65, reqid=client.4526.0:611, at_version=66'55, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.518964+0000", 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.96277027100000001, 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011483169999999999, 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.575 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518964+0000", 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518964+0000", 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518965+0000", 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4300000000000001e-07 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519750+0000", 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00078502599999999997 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519794+0000", 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4045999999999998e-05 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519801+0000", 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0949999999999998e-06 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519853+0000", 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2537000000000002e-05 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519865+0000", 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1507e-05 2024-09-15T02:49:11.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.530427+0000", 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010562531999999999 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.530447+0000", 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9483999999999998e-05 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.579 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.580 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:11.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:11.582 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:11.578+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:11.682 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:11.675+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:11.690 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:11.690 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:11.690 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:11.690 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:11.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:11.783 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:11.776+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:11.851 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:11.851 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:11.851 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:11.851 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:11.851 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:11.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:11.884 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:11.876+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:11.957 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:11.957 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:11.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:11.957 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:11.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:231 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.387327258999999, 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012963349000000001, 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 231 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565064+0000", 2024-09-15T02:49:11.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1140000000000002e-06 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565078+0000", 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3900000000000001e-05 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565120+0000", 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2070999999999998e-05 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565135+0000", 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5085999999999999e-05 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565263+0000", 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012826600000000001 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565272+0000", 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0259999999999996e-06 2024-09-15T02:49:11.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569405+0000", 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041323439999999996 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569560+0000", 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015537600000000001 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.570911+0000", 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013509349999999999 2024-09-15T02:49:11.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574145+0000", 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003233685 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.577972+0000", 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038275480000000001 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578024+0000", 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1997999999999999e-05 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:232 3.5s0 3:a906e5df:::smithi19125844-36:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.387223300999999, 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013333868, 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.964 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 232 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565166+0000", 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.054e-06 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565170+0000", 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1969999999999998e-06 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565205+0000", 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4829999999999997e-05 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565234+0000", 2024-09-15T02:49:11.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9320999999999999e-05 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565392+0000", 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015784999999999999 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565408+0000", 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5434000000000002e-05 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571539+0000", 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061311990000000004 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571658+0000", 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119486 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572348+0000", 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068956300000000002 2024-09-15T02:49:11.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574147+0000", 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017987459999999999 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578458+0000", 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043109690000000004 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578499+0000", 2024-09-15T02:49:11.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1219e-05 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:233 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.387143308999999, 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.49017684700000003, 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.971 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 233 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565246+0000", 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565250+0000", 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.76e-06 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565261+0000", 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1693e-05 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565268+0000", 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3570000000000003e-06 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565481+0000", 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021371699999999999 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565537+0000", 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5838000000000003e-05 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572781+0000", 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072440639999999997 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572928+0000", 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014673400000000001 2024-09-15T02:49:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.573788+0000", 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00086026199999999998 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582219+0000", 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084309350000000005 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055303+0000", 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.47308344099999999 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055422+0000", 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119119 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/33 ECSubWrite(tid=23, reqid=client.4526.0:236, at_version=39'18, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.378317704000001, 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089568029999999993, 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6399999999999998e-07 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574843+0000", 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00077236399999999999 2024-09-15T02:49:11.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575043+0000", 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019989399999999999 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575050+0000", 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.528e-06 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575124+0000", 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4041999999999993e-05 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575134+0000", 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582994+0000", 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078592579999999992 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.981 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.583027+0000", 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3747000000000001e-05 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 39/33 ECSubWriteReply(tid=18, last_complete=39'11, committed=1, applied=1))", 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.376131503, 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.480047051, 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:11.982 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:11.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576262+0000", 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.074e-06 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055023+0000", 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.478760992 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055037+0000", 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3607e-05 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055156+0000", 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118522 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.056304+0000", 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001148154 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:249 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~704512] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 26.366748045000001, 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031968222999999997, 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.986 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.987 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585643+0000", 2024-09-15T02:49:11.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7839999999999999e-06 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587905+0000", 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022617729999999999 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587945+0000", 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.981e-05 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587959+0000", 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4297000000000001e-05 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588041+0000", 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2514000000000003e-05 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588163+0000", 2024-09-15T02:49:11.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012119 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589878+0000", 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017156809999999999 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607051+0000", 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017173134999999999 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617495+0000", 2024-09-15T02:49:11.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010443661999999999 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617608+0000", 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113377 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:250 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~376832] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 26.363469934000001, 2024-09-15T02:49:11.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028844499999999999, 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 250 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588920+0000", 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.807e-06 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:11.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589853+0000", 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00093252000000000005 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589887+0000", 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4721000000000002e-05 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589910+0000", 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2787000000000001e-05 2024-09-15T02:49:11.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601237+0000", 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011326924 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601404+0000", 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016697000000000001 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.602323+0000", 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00091864900000000001 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607163+0000", 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048403409999999997 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617711+0000", 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010548119999999999 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617763+0000", 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1660999999999999e-05 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.998 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:307 3.5s0 3:ae24c343:::smithi19125844-3:head [copy-from ver 10] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.333159628000001, 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.021447352999999999, 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 307 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:11.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619232+0000", 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-15T02:49:12.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619263+0000", 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1090000000000002e-05 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619289+0000", 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5284000000000001e-05 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619300+0000", 2024-09-15T02:49:12.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1518000000000001e-05 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619367+0000", 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6983000000000006e-05 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619450+0000", 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2785000000000001e-05 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.630087+0000", 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010637047 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635664+0000", 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055774500000000003 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640607+0000", 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049425019999999997 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640676+0000", 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9078000000000005e-05 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.004 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:308 3.5s0 3:ae24c343:::smithi19125844-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.333069991999999, 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016310021000000001, 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 308 2024-09-15T02:49:12.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619319+0000", 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9500000000000002e-07 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619324+0000", 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.87e-06 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619332+0000", 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2539999999999992e-06 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619335+0000", 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0079999999999998e-06 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619592+0000", 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000256724 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619634+0000", 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2011999999999997e-05 2024-09-15T02:49:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635446+0000", 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015811935999999999 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635533+0000", 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7126000000000006e-05 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635628+0000", 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5495999999999995e-05 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.010 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.302298017999998, 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013723521000000001, 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650091+0000", 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04e-06 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650742+0000", 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000650436 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650792+0000", 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0680000000000003e-05 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650812+0000", 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9397000000000001e-05 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650905+0000", 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3294000000000005e-05 2024-09-15T02:49:12.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650922+0000", 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7209000000000001e-05 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663784+0000", 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012861490999999999 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663814+0000", 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9974000000000001e-05 2024-09-15T02:49:12.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.250618096, 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084791039999999995, 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701770+0000", 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5699999999999999e-07 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701896+0000", 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012543400000000001 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701967+0000", 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1172000000000002e-05 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.701978+0000", 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1e-05 2024-09-15T02:49:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702056+0000", 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7615999999999997e-05 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702069+0000", 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3746e-05 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710220+0000", 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081502959999999996 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710249+0000", 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9583000000000001e-05 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:12.019 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.676970955000002, 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092230960000000001, 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3300000000000001e-07 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276746+0000", 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013288589999999999 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276893+0000", 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000146849 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276906+0000", 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2347000000000001e-05 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276979+0000", 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3198e-05 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277005+0000", 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5837e-05 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284582+0000", 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075769779999999998 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284640+0000", 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8795000000000002e-05 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.025149759, 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009421937, 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927241+0000", 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0899999999999999e-06 2024-09-15T02:49:12.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927476+0000", 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000235047 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927608+0000", 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013218999999999999 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927627+0000", 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.965e-05 2024-09-15T02:49:12.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927797+0000", 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016982 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927817+0000", 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9939999999999999e-05 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936590+0000", 2024-09-15T02:49:12.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087725600000000004 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936660+0000", 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0640000000000001e-05 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:12.028 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.639648448999999, 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014263519000000001, 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312741+0000", 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1999999999999998e-07 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313098+0000", 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035754 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313251+0000", 2024-09-15T02:49:12.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000153177 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313275+0000", 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3365000000000001e-05 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313599+0000", 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032383899999999997 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313628+0000", 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.921e-05 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326410+0000", 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012781957 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.327003+0000", 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00059361099999999997 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 57/33 ECSubWrite(tid=39, reqid=client.4526.0:482, at_version=57'24, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.623400209, 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010144892000000001, 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:12.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328990+0000", 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7099999999999999e-06 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329132+0000", 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000142068 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329222+0000", 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0607999999999997e-05 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329240+0000", 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7524e-05 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329313+0000", 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2658999999999997e-05 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329323+0000", 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0497e-05 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339071+0000", 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097483840000000006 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339133+0000", 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1441999999999999e-05 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.5385625369999998, 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096704219999999997, 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7800000000000001e-07 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413927+0000", 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101127 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413988+0000", 2024-09-15T02:49:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1103000000000001e-05 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413997+0000", 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7709999999999995e-06 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414099+0000", 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102274 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414115+0000", 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6223e-05 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423433+0000", 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0093179969999999997 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423496+0000", 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2749000000000005e-05 2024-09-15T02:49:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/33 ECSubWrite(tid=54, reqid=client.4526.0:537, at_version=60'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.424373+0000", 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.5280153270000003, 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080799630000000008, 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424373+0000", 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424373+0000", 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424374+0000", 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.127e-06 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424544+0000", 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017042399999999999 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424607+0000", 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2649000000000003e-05 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424614+0000", 2024-09-15T02:49:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8109999999999997e-06 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424739+0000", 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012517600000000001 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.424749+0000", 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0293e-05 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432395+0000", 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076458740000000004 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.432453+0000", 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7608999999999997e-05 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.5s0 62/47 ECSubWrite(tid=315, reqid=osd.13.0:315, at_version=62'37, trim_to=0'0, roll_forward_to=61'35))", 2024-09-15T02:49:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:06.352484+0000", 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.5999046549999996, 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007324634, 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352484+0000", 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352484+0000", 2024-09-15T02:49:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352484+0000", 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1399999999999997e-07 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352544+0000", 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0260000000000002e-05 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352688+0000", 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014371699999999999 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352705+0000", 2024-09-15T02:49:12.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7062e-05 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352879+0000", 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017428300000000001 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.352905+0000", 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5694000000000002e-05 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.359738+0000", 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068325599999999997 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.359808+0000", 2024-09-15T02:49:12.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0444000000000005e-05 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 66/33 ECSubWrite(tid=66, reqid=client.4526.0:612, at_version=66'56, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.526476+0000", 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.425912163, 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0094503, 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526476+0000", 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526476+0000", 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526477+0000", 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.082e-06 2024-09-15T02:49:12.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526689+0000", 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021156699999999999 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526748+0000", 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9151000000000003e-05 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526757+0000", 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9209999999999998e-06 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526903+0000", 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014611 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526915+0000", 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1953000000000001e-05 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.535898+0000", 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089835350000000008 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.535926+0000", 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7980999999999999e-05 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.5s0 67/47 ECSubWrite(tid=349, reqid=osd.13.0:349, at_version=67'43, trim_to=0'0, roll_forward_to=66'39))", 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:11.419972+0000", 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.53241603999999998, 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.40161294400000003, 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419972+0000", 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419972+0000", 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419973+0000", 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3399999999999998e-07 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419998+0000", 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4856000000000001e-05 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819425+0000", 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.39942740799999998 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819438+0000", 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2401999999999999e-05 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819570+0000", 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000132423 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819593+0000", 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2605000000000001e-05 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.821532+0000", 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00193954 2024-09-15T02:49:12.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.821585+0000", 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2976000000000002e-05 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.063 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:12.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:12.065 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:12.062+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:12.166 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:12.160+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:12.172 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:12.172 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:12.172 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:12.173 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:12.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:12.267 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:12.262+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:12.282 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:12.283 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:12.283 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:12.283 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:12.283 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:12.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:12.369 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:12.366+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:12.394 INFO:tasks.rados.rados.0.smithi191.stdout:247: done (2 left) 2024-09-15T02:49:12.394 INFO:tasks.rados.rados.0.smithi191.stdout:248: copy_from oid 14 from oid 6 current snap is 18 2024-09-15T02:49:12.395 INFO:tasks.rados.rados.0.smithi191.stdout:245: expect (ObjNum 98 snap 15 seq_num 98) 2024-09-15T02:49:12.403 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:12.403 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:12.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:12.403 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:12.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:360 3.as0 3:54c1cad6:::smithi19125844-28:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.723360087, 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034523553999999998, 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 360 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675984+0000", 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.675987+0000", 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0900000000000001e-06 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676003+0000", 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6407000000000001e-05 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676045+0000", 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1390000000000002e-05 2024-09-15T02:49:12.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676065+0000", 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0157e-05 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676146+0000", 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0891999999999997e-05 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676153+0000", 2024-09-15T02:49:12.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0079999999999996e-06 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.678914+0000", 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002761262 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679005+0000", 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0648000000000004e-05 2024-09-15T02:49:12.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.693685+0000", 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014680723 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702650+0000", 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089650400000000005 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710453+0000", 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078028220000000001 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.710507+0000", 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4114999999999999e-05 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:361 3.as0 3:54c1cad6:::smithi19125844-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:12.410 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.723246505999999, 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026448546, 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 361 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676097+0000", 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676099+0000", 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.72e-06 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676105+0000", 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3729999999999998e-06 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676128+0000", 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2096999999999999e-05 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676139+0000", 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1603999999999999e-05 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676214+0000", 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4380000000000003e-05 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.676221+0000", 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8970000000000008e-06 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679152+0000", 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029307930000000001 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.679218+0000", 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5701e-05 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702293+0000", 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.02307476 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702424+0000", 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013107900000000001 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.702546+0000", 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000122142 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.145480252999999, 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.02916968, 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253864+0000", 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254277+0000", 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00041306699999999998 2024-09-15T02:49:12.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254344+0000", 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6806999999999999e-05 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254359+0000", 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4738e-05 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254477+0000", 2024-09-15T02:49:12.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118509 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254495+0000", 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7297999999999999e-05 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.282999+0000", 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028504345 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283033+0000", 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4436e-05 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s4 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:49:12.421 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.138758398, 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022480231, 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260585+0000", 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260587+0000", 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7400000000000001e-06 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260827+0000", 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000239459 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260907+0000", 2024-09-15T02:49:12.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9956999999999994e-05 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260923+0000", 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6592e-05 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268606+0000", 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076824390000000001 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268615+0000", 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9600000000000006e-06 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283052+0000", 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014437815 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283066+0000", 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3269e-05 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.425 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:472 3.as0 3:54c1cad6:::smithi19125844-28:head [copy-from ver 18 in=55b] snapc 13=[13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.07823892, 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.61688121200000001, 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.426 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 472 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321105+0000", 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321107+0000", 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9740000000000001e-06 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321117+0000", 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0214000000000001e-05 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321173+0000", 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5482000000000002e-05 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321185+0000", 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2028000000000001e-05 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321271+0000", 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6592e-05 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321500+0000", 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022896 2024-09-15T02:49:12.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927006+0000", 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.60550546299999997 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936994+0000", 2024-09-15T02:49:12.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009987915 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.937732+0000", 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073820699999999995 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.937986+0000", 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025437700000000002 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:473 3.as0 3:54c1cad6:::smithi19125844-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.078139141999999, 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.61562847600000004, 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 473 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.432 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321205+0000", 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321206+0000", 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2440000000000001e-06 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321211+0000", 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4580000000000001e-06 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321237+0000", 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5111999999999999e-05 2024-09-15T02:49:12.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321246+0000", 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1519999999999998e-06 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321773+0000", 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052693800000000002 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.321843+0000", 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0831e-05 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936316+0000", 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.61447244099999998 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936584+0000", 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026809300000000001 2024-09-15T02:49:12.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936833+0000", 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024920699999999999 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:481 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~753664] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.307419+0000", 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.091924998, 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.029233761, 2024-09-15T02:49:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 481 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.307419+0000", 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.307419+0000", 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.307422+0000", 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7930000000000002e-06 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.308924+0000", 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015024140000000001 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.308980+0000", 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6239999999999997e-05 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.309005+0000", 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4995999999999999e-05 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.309157+0000", 2024-09-15T02:49:12.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000151197 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.309361+0000", 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000203983 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312775+0000", 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034144829999999998 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336331+0000", 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023556128999999999 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336551+0000", 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000219865 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336653+0000", 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101661 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.442 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:482 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~720896] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312861+0000", 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.086482780000001, 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027088476, 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 482 2024-09-15T02:49:12.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312861+0000", 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312861+0000", 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312863+0000", 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2500000000000001e-06 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314314+0000", 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014503840000000001 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314363+0000", 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9240000000000003e-05 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314382+0000", 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8919e-05 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326033+0000", 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011651013 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326280+0000", 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024667800000000001 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.327782+0000", 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015028019999999999 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336793+0000", 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090105910000000001 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339895+0000", 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031024189999999999 2024-09-15T02:49:12.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339950+0000", 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.418e-05 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:483 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~49152] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.314412+0000", 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.08493142, 2024-09-15T02:49:12.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028623451000000001, 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 483 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314412+0000", 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314412+0000", 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314415+0000", 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3029999999999998e-06 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314530+0000", 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115097 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314549+0000", 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9020000000000001e-05 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314559+0000", 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6099999999999995e-06 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336836+0000", 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022277081000000001 2024-09-15T02:49:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336930+0000", 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4847999999999997e-05 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.337410+0000", 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047964500000000001 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339004+0000", 2024-09-15T02:49:12.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015934930000000001 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.342886+0000", 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038823569999999999 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343036+0000", 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014999699999999999 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:484 3.as0 3:54c1cad6:::smithi19125844-28:head [setxattr _header (57) in=7b] snapc 15=[15,14,13,11,10,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.314568+0000", 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.084775929999999, 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028727822, 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 484 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314568+0000", 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314568+0000", 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314569+0000", 2024-09-15T02:49:12.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.393e-06 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314574+0000", 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.454e-06 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314583+0000", 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0559999999999994e-06 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314588+0000", 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6200000000000004e-06 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338105+0000", 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023516358000000001 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338219+0000", 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011388899999999999 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338599+0000", 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038068599999999998 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339311+0000", 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000711672 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343168+0000", 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038569210000000001 2024-09-15T02:49:12.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343296+0000", 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127773 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:485 3.as0 3:54c1cad6:::smithi19125844-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.314594+0000", 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.084749886999999, 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034676314, 2024-09-15T02:49:12.459 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 485 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314594+0000", 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314594+0000", 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314595+0000", 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7999999999999993e-07 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314598+0000", 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1300000000000001e-06 2024-09-15T02:49:12.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314608+0000", 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4299999999999995e-06 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.314613+0000", 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.57e-06 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338862+0000", 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024249297 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.338991+0000", 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012851799999999999 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343772+0000", 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047815399999999999 2024-09-15T02:49:12.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.343870+0000", 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7619000000000002e-05 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.349270+0000", 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054002299999999998 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.320551+0000", 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.078792912000001, 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015703106000000001, 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320551+0000", 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320551+0000", 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320552+0000", 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3120000000000001e-06 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320564+0000", 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1736000000000001e-05 2024-09-15T02:49:12.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320615+0000", 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0899999999999997e-05 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320626+0000", 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0772e-05 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336239+0000", 2024-09-15T02:49:12.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015612862 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336254+0000", 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5523999999999999e-05 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/33 ECSubWriteReply(tid=38, last_complete=57'23, committed=1, applied=1))", 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.320765+0000", 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.078579155, 2024-09-15T02:49:12.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015630415000000002, 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320765+0000", 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320765+0000", 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320766+0000", 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7499999999999998e-07 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320772+0000", 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9070000000000004e-06 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320816+0000", 2024-09-15T02:49:12.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4338000000000001e-05 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.320827+0000", 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0915e-05 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336384+0000", 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015557034000000001 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.336395+0000", 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1246e-05 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 61/33 ECSubWriteReply(tid=62, last_complete=61'11, committed=1, applied=1))", 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.0456773269999999, 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.379230122, 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353667+0000", 2024-09-15T02:49:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.353672+0000", 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5719999999999996e-06 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732611+0000", 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.37893902499999998 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732624+0000", 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3166000000000001e-05 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732771+0000", 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014647299999999999 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.732897+0000", 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012614000000000001 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.475 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 62/33 ECSubWriteReply(tid=67, last_complete=62'13, committed=1, applied=1))", 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:06.371684+0000", 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.02765985, 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.66851037400000002, 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.371684+0000", 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.371684+0000", 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.371685+0000", 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1999999999999998e-07 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.371689+0000", 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.391e-06 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:07.040014+0000", 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.66832511699999997 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:07.040027+0000", 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2469999999999999e-05 2024-09-15T02:49:12.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:07.040124+0000", 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6749999999999994e-05 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:07.040194+0000", 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0826000000000003e-05 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:606 3.ds0 3:bcf25562:::smithi19125844-42:head [read 0~2326528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.500308+0000", 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.8990355559999998, 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015522614000000001, 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 606 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.500308+0000", 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.500308+0000", 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.500309+0000", 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1100000000000004e-07 2024-09-15T02:49:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.500315+0000", 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6860000000000003e-06 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.500330+0000", 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4973999999999999e-05 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.500347+0000", 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7482999999999998e-05 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.501925+0000", 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015775679999999999 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.502029+0000", 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010433 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.515831+0000", 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013801661999999999 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s6 66/33 ECSubWrite(tid=64, reqid=client.4526.0:610, at_version=66'54, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.511037+0000", 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.8883065110000001, 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015567286, 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.484 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511037+0000", 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511037+0000", 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511038+0000", 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8000000000000004e-07 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511998+0000", 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00095970699999999996 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512070+0000", 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2016999999999998e-05 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512079+0000", 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0469999999999999e-06 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512156+0000", 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7111000000000001e-05 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512172+0000", 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5401e-05 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526570+0000", 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014398143 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526605+0000", 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4980000000000001e-05 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:622 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~0] snapc 1d=[1d,1c,18,14,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.726458+0000", 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.67288624, 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015993817, 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 622 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726458+0000", 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726458+0000", 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726461+0000", 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1760000000000002e-06 2024-09-15T02:49:12.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726476+0000", 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5403000000000001e-05 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726511+0000", 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5197999999999997e-05 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726524+0000", 2024-09-15T02:49:12.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2211999999999999e-05 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726695+0000", 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017145100000000001 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726905+0000", 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000209645 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.728214+0000", 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013091120000000001 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.734698+0000", 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064844689999999996 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.742331+0000", 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076327119999999998 2024-09-15T02:49:12.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.742451+0000", 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000120439 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:623 3.as0 3:54c1cad6:::smithi19125844-28:head [rollback 11] snapc 1d=[1d,1c,18,14,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.726544+0000", 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.672799943, 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017165559, 2024-09-15T02:49:12.494 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 623 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726544+0000", 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726544+0000", 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726545+0000", 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8599999999999998e-07 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726550+0000", 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0980000000000001e-06 2024-09-15T02:49:12.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726564+0000", 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3998e-05 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726569+0000", 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9819999999999999e-06 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.730292+0000", 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003723311 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.730558+0000", 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026580799999999997 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.731804+0000", 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001246245 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.734701+0000", 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028974949999999999 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.743639+0000", 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089372040000000007 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.743710+0000", 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0832000000000002e-05 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.499 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:624 3.as0 3:54c1cad6:::smithi19125844-28:head [append 0~0] snapc 1d=[1d,1c,18,14,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.726574+0000", 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.672770061, 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017793039, 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:12.500 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 624 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726574+0000", 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:12.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726574+0000", 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726574+0000", 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6199999999999999e-07 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726577+0000", 2024-09-15T02:49:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3180000000000001e-06 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726586+0000", 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1409999999999994e-06 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.726590+0000", 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:12.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.732833+0000", 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0062425149999999997 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.733201+0000", 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036877200000000001 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.734144+0000", 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00094273100000000004 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.734964+0000", 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000820015 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.744261+0000", 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092964129999999999 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.744367+0000", 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106301 2024-09-15T02:49:12.505 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.506 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.506 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.506 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:12.506 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:12.506 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:12.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:244: finishing rollback tid 0 to smithi19125844-28 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:244: finishing rollback tid 1 to smithi19125844-28 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:244: finishing rollback tid 2 to smithi19125844-28 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 44 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:248: finishing copy_from racing read to smithi19125844-14 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:248: finishing copy_from to smithi19125844-14 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 31 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:244: done (2 left) 2024-09-15T02:49:12.509 INFO:tasks.rados.rados.0.smithi191.stdout:245: done (1 left) 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:248: done (0 left) 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:249: delete oid 3 current snap is 18 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:249: done (0 left) 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:250: read oid 2 snap -1 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:250: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 22 current snap is 18 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:251: seq_num 105 ranges {360448=606208,966656=688128,1654784=475136} 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:251: writing smithi19125844-22 from 360448 to 966656 tid 1 2024-09-15T02:49:12.510 INFO:tasks.rados.rados.0.smithi191.stdout:251: writing smithi19125844-22 from 966656 to 1654784 tid 2 2024-09-15T02:49:12.510 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:12.500+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:12.511 INFO:tasks.rados.rados.0.smithi191.stdout:251: writing smithi19125844-22 from 1654784 to 2129920 tid 3 2024-09-15T02:49:12.511 INFO:tasks.rados.rados.0.smithi191.stdout:252: snap_create 2024-09-15T02:49:12.511 INFO:tasks.rados.rados.0.smithi191.stdout:250: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-15T02:49:12.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:12.579 INFO:tasks.rados.rados.0.smithi191.stdout:251: finishing write tid 1 to smithi19125844-22 2024-09-15T02:49:12.579 INFO:tasks.rados.rados.0.smithi191.stdout:251: finishing write tid 2 to smithi19125844-22 2024-09-15T02:49:12.608 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:12.603+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:12.611 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:12.611 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:12.611 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:12.611 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:12.620 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:12.679 INFO:tasks.rados.rados.0.smithi191.stdout:251: finishing write tid 3 to smithi19125844-22 2024-09-15T02:49:12.698 INFO:tasks.rados.rados.0.smithi191.stdout:251: finishing write tid 4 to smithi19125844-22 2024-09-15T02:49:12.701 INFO:tasks.rados.rados.0.smithi191.stdout:251: finishing write tid 5 to smithi19125844-22 2024-09-15T02:49:12.701 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 22 v 72 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-15T02:49:12.701 INFO:tasks.rados.rados.0.smithi191.stdout:251: left oid 22 (ObjNum 105 snap 18 seq_num 105) 2024-09-15T02:49:12.709 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:12.704+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:12.719 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:12.719 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:12.719 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:12.719 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:12.719 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:12.725 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:12.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:12.754 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:12.784 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:12.809 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:12.804+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:12.813 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:12.824 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:12.824 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:12.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:12.840 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:12.869 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:12.898 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:12.910 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:12.904+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:12.925 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:12.930 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:12.930 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:12.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:12.952 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:12.982 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:13.010 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:13.006+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:13.011 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:13.040 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:13.041 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.041 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:13.066 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:13.093 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:13.112 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:13.104+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:13.122 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:13.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:13.115+0000 7fe9a37fe700 0 -- 172.21.15.6:0/2126001637 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fe9a4198740 msgr2=0x7fe9a419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:13.151 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.151 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.160 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:13.213 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:13.208+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:13.259 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.259 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:13.314 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:13.310+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:13.356 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:13.360 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:13.360 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:13.360 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:13.360 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:13.367 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.367 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:13.410 INFO:tasks.rados.rados.0.smithi191.stdout:250: done (2 left) 2024-09-15T02:49:13.411 INFO:tasks.rados.rados.0.smithi191.stdout:251: done (1 left) 2024-09-15T02:49:13.411 INFO:tasks.rados.rados.0.smithi191.stdout:252: done (0 left) 2024-09-15T02:49:13.411 INFO:tasks.rados.rados.0.smithi191.stdout:253: snap_remove snap 10 2024-09-15T02:49:13.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:49:13.424 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:13.420+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:13.458 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 10] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:49:13.459 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:49:13.459 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-15T02:49:13.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:49:13.474 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.474 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:13.515 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:13.508+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:13.580 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.581 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:13.616 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:13.612+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:13.687 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.688 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:13.718 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:13.712+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:13.760 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:49:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:49:13.776 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-15T02:49:13.402788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67125811,"num_objects":106,"num_object_clones":54,"num_object_copies":722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":11,"num_read":429,"num_read_kb":130486,"num_write":664,"num_write_kb":76882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":21479424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":776,"ondisk_log_size":776,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":4884548,"kb_used_data":126320,"kb_used_omap":26,"kb_used_meta":4757989,"kb_avail":1505064892,"statfs":{"total":1546188226560,"available":1541186449408,"internally_reserved":0,"allocated":129351680,"data_stored":164627831,"data_compressed":1407298,"data_compressed_allocated":3002368,"data_compressed_original":6897664,"omap_allocated":27344,"internal_metadata":4872181040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2932736,"num_objects":6,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":39,"num_read_kb":42602,"num_write":37,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042049"},"pg_stats":[{"pgid":"3.f","version":"67'43","reported_seq":179,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.784373+0000","last_change":"2024-09-15T02:49:11.786186+0000","last_active":"2024-09-15T02:49:11.784373+0000","last_peered":"2024-09-15T02:49:11.784373+0000","last_clean":"2024-09-15T02:49:11.784373+0000","last_became_active":"2024-09-15T02:48:58.322126+0000","last_became_peered":"2024-09-15T02:48:58.322126+0000","last_unstale":"2024-09-15T02:49:11.784373+0000","last_undegraded":"2024-09-15T02:49:11.784373+0000","last_fullsized":"2024-09-15T02:49:11.784373+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:03.904159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.39316041800000001,"stat_sum":{"num_bytes":2703360,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":6825,"num_write":36,"num_write_kb":2640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.e","version":"67'71","reported_seq":175,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.407927+0000","last_change":"2024-09-15T02:49:11.408892+0000","last_active":"2024-09-15T02:49:11.407927+0000","last_peered":"2024-09-15T02:49:11.407927+0000","last_clean":"2024-09-15T02:49:11.407927+0000","last_became_active":"2024-09-15T02:48:38.548379+0000","last_became_peered":"2024-09-15T02:48:38.548379+0000","last_unstale":"2024-09-15T02:49:11.407927+0000","last_undegraded":"2024-09-15T02:49:11.407927+0000","last_fullsized":"2024-09-15T02:49:11.407927+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:04.803440+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.01597935,"stat_sum":{"num_bytes":5603328,"num_objects":12,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":26,"num_read_kb":9419,"num_write":115,"num_write_kb":6800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.d","version":"67'15","reported_seq":120,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.396571+0000","last_change":"2024-09-15T02:49:11.397490+0000","last_active":"2024-09-15T02:49:11.396571+0000","last_peered":"2024-09-15T02:49:11.396571+0000","last_clean":"2024-09-15T02:49:11.396571+0000","last_became_active":"2024-09-15T02:48:38.545950+0000","last_became_peered":"2024-09-15T02:48:38.545950+0000","last_unstale":"2024-09-15T02:49:11.396571+0000","last_undegraded":"2024-09-15T02:49:11.396571+0000","last_fullsized":"2024-09-15T02:49:11.396571+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:58.729969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011192554,"stat_sum":{"num_bytes":2883584,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":8885,"num_write":8,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.c","version":"59'13","reported_seq":134,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.392987+0000","last_change":"2024-09-15T02:49:11.393345+0000","last_active":"2024-09-15T02:49:11.392987+0000","last_peered":"2024-09-15T02:49:11.392987+0000","last_clean":"2024-09-15T02:49:11.392987+0000","last_became_active":"2024-09-15T02:48:58.321514+0000","last_became_peered":"2024-09-15T02:48:58.321514+0000","last_unstale":"2024-09-15T02:49:11.392987+0000","last_undegraded":"2024-09-15T02:49:11.392987+0000","last_fullsized":"2024-09-15T02:49:11.392987+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:53.632511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056526799999999996,"stat_sum":{"num_bytes":1114112,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1949696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.b","version":"67'59","reported_seq":160,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.420708+0000","last_change":"2024-09-15T02:49:11.421839+0000","last_active":"2024-09-15T02:49:11.420708+0000","last_peered":"2024-09-15T02:49:11.420708+0000","last_clean":"2024-09-15T02:49:11.420708+0000","last_became_active":"2024-09-15T02:48:38.556657+0000","last_became_peered":"2024-09-15T02:48:38.556657+0000","last_unstale":"2024-09-15T02:49:11.420708+0000","last_undegraded":"2024-09-15T02:49:11.420708+0000","last_fullsized":"2024-09-15T02:49:11.420708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:46.112890+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.028150261999999999,"stat_sum":{"num_bytes":3604480,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":22,"num_read_kb":2025,"num_write":47,"num_write_kb":5344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,7],"acting":[13,1,11,15,6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.a","version":"67'46","reported_seq":147,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.396539+0000","last_change":"2024-09-15T02:49:11.397273+0000","last_active":"2024-09-15T02:49:11.396539+0000","last_peered":"2024-09-15T02:49:11.396539+0000","last_clean":"2024-09-15T02:49:11.396539+0000","last_became_active":"2024-09-15T02:48:38.549833+0000","last_became_peered":"2024-09-15T02:48:38.549833+0000","last_unstale":"2024-09-15T02:49:11.396539+0000","last_undegraded":"2024-09-15T02:49:11.396539+0000","last_fullsized":"2024-09-15T02:49:11.396539+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:57.227388+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011346172,"stat_sum":{"num_bytes":6553600,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":18,"num_read_kb":8071,"num_write":19,"num_write_kb":5712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,6,8],"acting":[12,4,7,3,14,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.9","version":"61'43","reported_seq":193,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.387847+0000","last_change":"2024-09-15T02:49:11.388210+0000","last_active":"2024-09-15T02:49:11.387847+0000","last_peered":"2024-09-15T02:49:11.387847+0000","last_clean":"2024-09-15T02:49:11.387847+0000","last_became_active":"2024-09-15T02:48:58.324925+0000","last_became_peered":"2024-09-15T02:48:58.324925+0000","last_unstale":"2024-09-15T02:49:11.387847+0000","last_undegraded":"2024-09-15T02:49:11.387847+0000","last_fullsized":"2024-09-15T02:49:11.387847+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:04.858136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071469100000000005,"stat_sum":{"num_bytes":3883008,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":16716,"num_write":23,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4423680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.8","version":"67'38","reported_seq":147,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.408001+0000","last_change":"2024-09-15T02:49:11.408894+0000","last_active":"2024-09-15T02:49:11.408001+0000","last_peered":"2024-09-15T02:49:11.408001+0000","last_clean":"2024-09-15T02:49:11.408001+0000","last_became_active":"2024-09-15T02:48:38.549625+0000","last_became_peered":"2024-09-15T02:48:38.549625+0000","last_unstale":"2024-09-15T02:49:11.408001+0000","last_undegraded":"2024-09-15T02:49:11.408001+0000","last_fullsized":"2024-09-15T02:49:11.408001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:42.659479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013369750999999999,"stat_sum":{"num_bytes":7028736,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":8217,"num_write":24,"num_write_kb":7760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":187,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-15T02:49:10.375800+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:49:10.375800+0000","last_peered":"2024-09-15T02:49:10.375800+0000","last_clean":"2024-09-15T02:49:10.375800+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:49:10.375800+0000","last_undegraded":"2024-09-15T02:49:10.375800+0000","last_fullsized":"2024-09-15T02:49:10.375800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.005063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":54,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-15T02:49:10.377420+0000","last_change":"2024-09-15T02:48:29.636494+0000","last_active":"2024-09-15T02:49:10.377420+0000","last_peered":"2024-09-15T02:49:10.377420+0000","last_clean":"2024-09-15T02:49:10.377420+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:49:10.377420+0000","last_undegraded":"2024-09-15T02:49:10.377420+0000","last_fullsized":"2024-09-15T02:49:10.377420+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:30.837572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"67'35","reported_seq":130,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.406690+0000","last_change":"2024-09-15T02:49:11.408249+0000","last_active":"2024-09-15T02:49:11.406690+0000","last_peered":"2024-09-15T02:49:11.406690+0000","last_clean":"2024-09-15T02:49:11.406690+0000","last_became_active":"2024-09-15T02:48:38.545032+0000","last_became_peered":"2024-09-15T02:48:38.545032+0000","last_unstale":"2024-09-15T02:49:11.406690+0000","last_undegraded":"2024-09-15T02:49:11.406690+0000","last_fullsized":"2024-09-15T02:49:11.406690+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:01.938948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012934212,"stat_sum":{"num_bytes":3964928,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":5,"num_write":56,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T02:49:07.377470+0000","last_change":"2024-09-15T02:48:52.652217+0000","last_active":"2024-09-15T02:49:07.377470+0000","last_peered":"2024-09-15T02:49:07.377470+0000","last_clean":"2024-09-15T02:49:07.377470+0000","last_became_active":"2024-09-15T02:48:52.651825+0000","last_became_peered":"2024-09-15T02:48:52.651825+0000","last_unstale":"2024-09-15T02:49:07.377470+0000","last_undegraded":"2024-09-15T02:49:07.377470+0000","last_fullsized":"2024-09-15T02:49:07.377470+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:38.214912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"67'53","reported_seq":201,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.411515+0000","last_change":"2024-09-15T02:49:11.412113+0000","last_active":"2024-09-15T02:49:11.411515+0000","last_peered":"2024-09-15T02:49:11.411515+0000","last_clean":"2024-09-15T02:49:11.411515+0000","last_became_active":"2024-09-15T02:48:58.327894+0000","last_became_peered":"2024-09-15T02:48:58.327894+0000","last_unstale":"2024-09-15T02:49:11.411515+0000","last_undegraded":"2024-09-15T02:49:11.411515+0000","last_fullsized":"2024-09-15T02:49:11.411515+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:38.445276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018026858999999999,"stat_sum":{"num_bytes":6668288,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":39,"num_read_kb":19598,"num_write":27,"num_write_kb":6512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11059200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-15T02:49:10.376877+0000","last_change":"2024-09-15T02:48:29.420976+0000","last_active":"2024-09-15T02:49:10.376877+0000","last_peered":"2024-09-15T02:49:10.376877+0000","last_clean":"2024-09-15T02:49:10.376877+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:49:10.376877+0000","last_undegraded":"2024-09-15T02:49:10.376877+0000","last_fullsized":"2024-09-15T02:49:10.376877+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:33.874284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"67'26","reported_seq":134,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.393498+0000","last_change":"2024-09-15T02:49:11.394140+0000","last_active":"2024-09-15T02:49:11.393498+0000","last_peered":"2024-09-15T02:49:11.393498+0000","last_clean":"2024-09-15T02:49:11.393498+0000","last_became_active":"2024-09-15T02:48:38.545651+0000","last_became_peered":"2024-09-15T02:48:38.545651+0000","last_unstale":"2024-09-15T02:49:11.393498+0000","last_undegraded":"2024-09-15T02:49:11.393498+0000","last_fullsized":"2024-09-15T02:49:11.393498+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:04.452887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006677924,"stat_sum":{"num_bytes":3276800,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15319,"num_write":12,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":45,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-15T02:49:10.374396+0000","last_change":"2024-09-15T02:48:52.654051+0000","last_active":"2024-09-15T02:49:10.374396+0000","last_peered":"2024-09-15T02:49:10.374396+0000","last_clean":"2024-09-15T02:49:10.374396+0000","last_became_active":"2024-09-15T02:48:52.653285+0000","last_became_peered":"2024-09-15T02:48:52.653285+0000","last_unstale":"2024-09-15T02:49:10.374396+0000","last_undegraded":"2024-09-15T02:49:10.374396+0000","last_fullsized":"2024-09-15T02:49:10.374396+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:44.752359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"67'63","reported_seq":165,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.417390+0000","last_change":"2024-09-15T02:49:11.418347+0000","last_active":"2024-09-15T02:49:11.417390+0000","last_peered":"2024-09-15T02:49:11.417390+0000","last_clean":"2024-09-15T02:49:11.417390+0000","last_became_active":"2024-09-15T02:48:38.509662+0000","last_became_peered":"2024-09-15T02:48:38.509662+0000","last_unstale":"2024-09-15T02:49:11.417390+0000","last_undegraded":"2024-09-15T02:49:11.417390+0000","last_fullsized":"2024-09-15T02:49:11.417390+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:58.357500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016350678,"stat_sum":{"num_bytes":5144576,"num_objects":10,"num_object_clones":6,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":24,"num_read_kb":3177,"num_write":43,"num_write_kb":5024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"3.5","version":"67'43","reported_seq":151,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.823365+0000","last_change":"2024-09-15T02:49:11.825090+0000","last_active":"2024-09-15T02:49:11.823365+0000","last_peered":"2024-09-15T02:49:11.823365+0000","last_clean":"2024-09-15T02:49:11.823365+0000","last_became_active":"2024-09-15T02:48:52.655868+0000","last_became_peered":"2024-09-15T02:48:52.655868+0000","last_unstale":"2024-09-15T02:49:11.823365+0000","last_undegraded":"2024-09-15T02:49:11.823365+0000","last_fullsized":"2024-09-15T02:49:11.823365+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:04.964727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.43059737799999998,"stat_sum":{"num_bytes":3276800,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":5,"num_write":21,"num_write_kb":4768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-15T02:49:10.382187+0000","last_change":"2024-09-15T02:48:29.823846+0000","last_active":"2024-09-15T02:49:10.382187+0000","last_peered":"2024-09-15T02:49:10.382187+0000","last_clean":"2024-09-15T02:49:10.382187+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:49:10.382187+0000","last_undegraded":"2024-09-15T02:49:10.382187+0000","last_fullsized":"2024-09-15T02:49:10.382187+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:26.621594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"36'2","reported_seq":102,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.401181+0000","last_change":"2024-09-15T02:49:11.401583+0000","last_active":"2024-09-15T02:49:11.401181+0000","last_peered":"2024-09-15T02:49:11.401181+0000","last_clean":"2024-09-15T02:49:11.401181+0000","last_became_active":"2024-09-15T02:48:38.623141+0000","last_became_peered":"2024-09-15T02:48:38.623141+0000","last_unstale":"2024-09-15T02:49:11.401181+0000","last_undegraded":"2024-09-15T02:49:11.401181+0000","last_fullsized":"2024-09-15T02:49:11.401181+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:00.649077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060187800000000003,"stat_sum":{"num_bytes":573440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1682,"num_write":2,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T02:49:07.377704+0000","last_change":"2024-09-15T02:48:29.419531+0000","last_active":"2024-09-15T02:49:07.377704+0000","last_peered":"2024-09-15T02:49:07.377704+0000","last_clean":"2024-09-15T02:49:07.377704+0000","last_became_active":"2024-09-15T02:48:28.248628+0000","last_became_peered":"2024-09-15T02:48:28.248628+0000","last_unstale":"2024-09-15T02:49:07.377704+0000","last_undegraded":"2024-09-15T02:49:07.377704+0000","last_fullsized":"2024-09-15T02:49:07.377704+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:53.626378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055936099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"67'86","reported_seq":209,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.826354+0000","last_change":"2024-09-15T02:49:11.828464+0000","last_active":"2024-09-15T02:49:11.826354+0000","last_peered":"2024-09-15T02:49:11.826354+0000","last_clean":"2024-09-15T02:49:11.826354+0000","last_became_active":"2024-09-15T02:48:38.550790+0000","last_became_peered":"2024-09-15T02:48:38.550790+0000","last_unstale":"2024-09-15T02:49:11.826354+0000","last_undegraded":"2024-09-15T02:49:11.826354+0000","last_fullsized":"2024-09-15T02:49:11.826354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:03.358479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.433290073,"stat_sum":{"num_bytes":8667136,"num_objects":12,"num_object_clones":6,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":54,"num_read_kb":25749,"num_write":47,"num_write_kb":10464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,1,11,5,0],"acting":[13,6,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-15T02:49:10.374798+0000","last_change":"2024-09-15T02:48:29.426846+0000","last_active":"2024-09-15T02:49:10.374798+0000","last_peered":"2024-09-15T02:49:10.374798+0000","last_clean":"2024-09-15T02:49:10.374798+0000","last_became_active":"2024-09-15T02:48:27.400822+0000","last_became_peered":"2024-09-15T02:48:27.400822+0000","last_unstale":"2024-09-15T02:49:10.374798+0000","last_undegraded":"2024-09-15T02:49:10.374798+0000","last_fullsized":"2024-09-15T02:49:10.374798+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:32.325018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"67'25","reported_seq":149,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.417503+0000","last_change":"2024-09-15T02:49:11.418210+0000","last_active":"2024-09-15T02:49:11.417503+0000","last_peered":"2024-09-15T02:49:11.417503+0000","last_clean":"2024-09-15T02:49:11.417503+0000","last_became_active":"2024-09-15T02:48:58.325506+0000","last_became_peered":"2024-09-15T02:48:58.325506+0000","last_unstale":"2024-09-15T02:49:11.417503+0000","last_undegraded":"2024-09-15T02:49:11.417503+0000","last_fullsized":"2024-09-15T02:49:11.417503+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:48.747639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024072550000000002,"stat_sum":{"num_bytes":1589248,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":15,"num_read_kb":4718,"num_write":30,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2785280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"14","length":"2"},{"start":"1b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":32,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-15T02:49:11.399006+0000","last_change":"2024-09-15T02:48:52.652454+0000","last_active":"2024-09-15T02:49:11.399006+0000","last_peered":"2024-09-15T02:49:11.399006+0000","last_clean":"2024-09-15T02:49:11.399006+0000","last_became_active":"2024-09-15T02:48:52.652176+0000","last_became_peered":"2024-09-15T02:48:52.652176+0000","last_unstale":"2024-09-15T02:49:11.399006+0000","last_undegraded":"2024-09-15T02:49:11.399006+0000","last_fullsized":"2024-09-15T02:49:11.399006+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:56.216530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":66535424,"num_objects":102,"num_object_clones":54,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":11,"num_read":343,"num_read_kb":130413,"num_write":517,"num_write_kb":74208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":21479424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117297152,"data_stored":155164672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":661,"ondisk_log_size":661,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":22,"seq":94489280523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306916,"kb_used_data":9560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94064924,"statfs":{"total":96636764160,"available":96322482176,"internally_reserved":0,"allocated":9789440,"data_stored":12004404,"data_compressed":81450,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":485.07400000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]}]},{"osd":7,"up_from":22,"seq":94489280526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306172,"kb_used_data":8812,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065668,"statfs":{"total":96636764160,"available":96323244032,"internally_reserved":0,"allocated":9023488,"data_stored":11542402,"data_compressed":89351,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,6,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":4,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308540,"kb_used_data":11180,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94063300,"statfs":{"total":96636764160,"available":96320819200,"internally_reserved":0,"allocated":11448320,"data_stored":15171458,"data_compressed":89351,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]}]},{"osd":5,"up_from":21,"seq":90194313229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303876,"kb_used_data":6516,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94067964,"statfs":{"total":96636764160,"available":96325595136,"internally_reserved":0,"allocated":6672384,"data_stored":8238315,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307532,"kb_used_data":10176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94064308,"statfs":{"total":96636764160,"available":96321851392,"internally_reserved":0,"allocated":10420224,"data_stored":13559019,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]}]},{"osd":13,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307332,"kb_used_data":9976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94064508,"statfs":{"total":96636764160,"available":96322056192,"internally_reserved":0,"allocated":10215424,"data_stored":13215533,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6950000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1429999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9219999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.74}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.286}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.891}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":8572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":8777728,"data_stored":11081517,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304344,"kb_used_data":6988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94067496,"statfs":{"total":96636764160,"available":96325115904,"internally_reserved":0,"allocated":7155712,"data_stored":8685357,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7629999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]}]},{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306672,"kb_used_data":9308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065168,"statfs":{"total":96636764160,"available":96322732032,"internally_reserved":0,"allocated":9531392,"data_stored":13005048,"data_compressed":93005,"data_compressed_allocated":212992,"data_compressed_original":872448,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.748}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1040000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]}]},{"osd":15,"up_from":22,"seq":94489280524,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303416,"kb_used_data":6060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068424,"statfs":{"total":96636764160,"available":96326066176,"internally_reserved":0,"allocated":6205440,"data_stored":8136493,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303052,"kb_used_data":5696,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068788,"statfs":{"total":96636764160,"available":96326438912,"internally_reserved":0,"allocated":5832704,"data_stored":7407405,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.631}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.609}]}]},{"osd":8,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303636,"kb_used_data":6280,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068204,"statfs":{"total":96636764160,"available":96325840896,"internally_reserved":0,"allocated":6430720,"data_stored":8308544,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":640.51300000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301048,"kb_used_data":3692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070792,"statfs":{"total":96636764160,"available":96328491008,"internally_reserved":0,"allocated":3780608,"data_stored":4756734,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]}]},{"osd":10,"up_from":42,"seq":180388626439,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303784,"kb_used_data":5916,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94068056,"statfs":{"total":96636764160,"available":96325689344,"internally_reserved":0,"allocated":6057984,"data_stored":5881560,"data_compressed":85400,"data_compressed_allocated":180224,"data_compressed_original":360448,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280525,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310552,"kb_used_data":13196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94061288,"statfs":{"total":96636764160,"available":96318758912,"internally_reserved":0,"allocated":13512704,"data_stored":18295117,"data_compressed":94981,"data_compressed_allocated":217088,"data_compressed_original":880640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5860000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.748}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7849999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280524,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":4497408,"data_stored":5338925,"data_compressed":87376,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9674752,"data_stored":13041664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8028160,"data_stored":10563584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8740864,"data_stored":11927552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5083136,"data_stored":6889472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10686464,"data_stored":14643200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5926912,"data_stored":7720960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9080832,"data_stored":11517952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8261632,"data_stored":11014144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5677056,"data_stored":7790592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3031040,"data_stored":4239360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5320704,"data_stored":5373952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12709888,"data_stored":17248256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3747840,"data_stored":4820992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":9465856,"data_stored":12697600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":6406144,"data_stored":8056832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5455872,"data_stored":7618560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:49:13.779 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-15T02:49:13.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-15T02:49:13.792 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.792 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.801 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:13.818 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:13.811+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:13.911 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:13.911 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:13.920 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:13.916+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:13.921 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:14.018 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.018 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.021 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:14.016+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:14.027 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:14.122 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:14.118+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:14.122 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.122 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.131 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:14.223 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:14.216+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:14.232 INFO:tasks.rados.rados.0.smithi191.stdout:253: done (0 left) 2024-09-15T02:49:14.232 INFO:tasks.rados.rados.0.smithi191.stdout:254: copy_from oid 11 from oid 16 current snap is 19 2024-09-15T02:49:14.232 INFO:tasks.rados.rados.0.smithi191.stdout:255: rollback oid 40 current snap is 19 2024-09-15T02:49:14.233 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 40 to 14 2024-09-15T02:49:14.233 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 21 current snap is 19 2024-09-15T02:49:14.233 INFO:tasks.rados.rados.0.smithi191.stdout:256: seq_num 106 ranges {344064=622592,966656=442368,1409024=524288,1933312=196608} 2024-09-15T02:49:14.234 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.235 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.240 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:49:14.243 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 3 pg_num to 26 2024-09-15T02:49:14.246 INFO:tasks.rados.rados.0.smithi191.stdout:256: writing smithi19125844-21 from 344064 to 966656 tid 1 2024-09-15T02:49:14.249 INFO:tasks.rados.rados.0.smithi191.stdout:256: writing smithi19125844-21 from 966656 to 1409024 tid 2 2024-09-15T02:49:14.251 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:49:14.253 INFO:tasks.rados.rados.0.smithi191.stdout:256: writing smithi19125844-21 from 1409024 to 1933312 tid 3 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:256: writing smithi19125844-21 from 1933312 to 2129920 tid 4 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:257: rollback oid 4 current snap is 19 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 4 to 18 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:254: finishing copy_from to smithi19125844-11 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:254: got expected ENOENT (src dne) 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:254: finishing copy_from racing read to smithi19125844-11 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:254: done (3 left) 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:258: copy_from oid 50 from oid 23 current snap is 19 2024-09-15T02:49:14.256 INFO:tasks.rados.rados.0.smithi191.stdout:259: delete oid 33 current snap is 19 2024-09-15T02:49:14.261 INFO:tasks.rados.rados.0.smithi191.stdout:255: finishing rollback tid 1 to smithi19125844-40 2024-09-15T02:49:14.261 INFO:tasks.rados.rados.0.smithi191.stdout:255: finishing rollback tid 2 to smithi19125844-40 2024-09-15T02:49:14.261 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 40 v 61 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-15T02:49:14.272 INFO:tasks.rados.rados.0.smithi191.stdout:257: finishing rollback tid 0 to smithi19125844-4 2024-09-15T02:49:14.273 INFO:tasks.rados.rados.0.smithi191.stdout:257: finishing rollback tid 1 to smithi19125844-4 2024-09-15T02:49:14.274 INFO:tasks.rados.rados.0.smithi191.stdout:257: finishing rollback tid 2 to smithi19125844-4 2024-09-15T02:49:14.274 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 4 v 98 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T02:49:14.276 INFO:tasks.rados.rados.0.smithi191.stdout:255: done (4 left) 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:257: done (3 left) 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:259: done (2 left) 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:260: read oid 30 snap 17 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:260: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:261: read oid 18 snap 5 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:261: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-15T02:49:14.277 INFO:tasks.rados.rados.0.smithi191.stdout:262: snap_remove snap 4 2024-09-15T02:49:14.324 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:14.316+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:14.344 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.345 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.354 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:49:14.424 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:14.422+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:14.449 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.449 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.458 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:14.526 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:14.520+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:14.561 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.561 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.570 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:14.626 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:14.622+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:14.672 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.672 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.681 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:14.728 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:14.722+0000 7fe707667700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22068) UID: 0 2024-09-15T02:49:14.777 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.777 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:14.827 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:14.824+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:14.895 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:14.895 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:14.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:14.929 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:14.923+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:15.009 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.010 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:15.029 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:15.024+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:15.115 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.115 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:15.130 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:15.126+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:15.231 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:15.224+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:15.236 INFO:tasks.rados.rados.0.smithi191.stdout:262: done (4 left) 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:263: snap_create 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:256: finishing write tid 1 to smithi19125844-21 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:258: finishing copy_from to smithi19125844-50 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:258: got expected ENOENT (src dne) 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:258: finishing copy_from racing read to smithi19125844-50 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:256: finishing write tid 2 to smithi19125844-21 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:256: finishing write tid 3 to smithi19125844-21 2024-09-15T02:49:15.237 INFO:tasks.rados.rados.0.smithi191.stdout:261: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-15T02:49:15.287 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.287 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:15.296 INFO:tasks.rados.rados.0.smithi191.stdout:256: finishing write tid 4 to smithi19125844-21 2024-09-15T02:49:15.297 INFO:tasks.rados.rados.0.smithi191.stdout:256: finishing write tid 5 to smithi19125844-21 2024-09-15T02:49:15.297 INFO:tasks.rados.rados.0.smithi191.stdout:260: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-15T02:49:15.311 INFO:tasks.rados.rados.0.smithi191.stdout:256: finishing write tid 6 to smithi19125844-21 2024-09-15T02:49:15.311 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 21 v 69 (ObjNum 106 snap 19 seq_num 106) dirty exists 2024-09-15T02:49:15.311 INFO:tasks.rados.rados.0.smithi191.stdout:256: left oid 21 (ObjNum 106 snap 19 seq_num 106) 2024-09-15T02:49:15.331 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:15.324+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:15.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:15.384 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.384 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:15.437 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:15.432+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:15.534 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:15.527+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:15.536 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:15.538 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.538 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:15.566 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:15.596 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:15.622 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:15.635 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:15.630+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:15.649 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:15.651 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.651 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:15.678 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:15.706 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:15.734 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:15.736 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:15.734+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:15.759 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:15.760 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:15.762 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:15.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:15.790 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:15.819 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:15.838 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:15.832+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:15.847 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:15.873 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:15.878 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.509832031000002, 2024-09-15T02:49:15.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:15.879 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.879 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:15.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:15.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:15.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.150260517, 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.887 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:15.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:15.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:15.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:15.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:15.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:15.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:15.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.541670523000001, 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018893783000000001, 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331907+0000", 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331908+0000", 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.133e-06 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331914+0000", 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5799999999999999e-06 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331925+0000", 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0893999999999999e-05 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331931+0000", 2024-09-15T02:49:15.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2199999999999997e-06 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342666+0000", 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01073519 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342753+0000", 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7175999999999993e-05 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.342888+0000", 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013462700000000001 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348168+0000", 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005280285 2024-09-15T02:49:15.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350557+0000", 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023890309999999998 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350698+0000", 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140755 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.350801+0000", 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102892 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.541640053000002, 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026305674000000001, 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.903 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:15.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331943+0000", 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7559999999999999e-06 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331955+0000", 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331964+0000", 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4069999999999993e-06 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343483+0000", 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011518995000000001 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343581+0000", 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8568999999999998e-05 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343713+0000", 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013195700000000001 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348183+0000", 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044698639999999996 2024-09-15T02:49:15.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358029+0000", 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0098460219999999994 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358143+0000", 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114335 2024-09-15T02:49:15.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358243+0000", 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9881000000000001e-05 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:15.909 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.541607376000002, 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026584729000000001, 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:15.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331971+0000", 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332135+0000", 2024-09-15T02:49:15.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000164098 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332158+0000", 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332165+0000", 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8669999999999996e-06 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344120+0000", 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011955222 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344205+0000", 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4895000000000006e-05 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344336+0000", 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130844 2024-09-15T02:49:15.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.349645+0000", 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053089770000000003 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358356+0000", 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087112079999999998 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358462+0000", 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105664 2024-09-15T02:49:15.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358555+0000", 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3110999999999997e-05 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.541397691, 2024-09-15T02:49:15.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:49:15.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:49:15.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.304925851, 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017901001, 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 98 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.922 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568652+0000", 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568653+0000", 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4700000000000001e-07 2024-09-15T02:49:15.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568658+0000", 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.006e-06 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568666+0000", 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5369999999999997e-06 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568671+0000", 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3700000000000003e-06 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580006+0000", 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011334946 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580064+0000", 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7207999999999998e-05 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580154+0000", 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0682999999999999e-05 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.582087+0000", 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019328520000000001 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586462+0000", 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004374452 2024-09-15T02:49:15.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586503+0000", 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1872000000000002e-05 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586553+0000", 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9327999999999999e-05 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.927 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.304823747, 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017954067000000001, 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 99 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568754+0000", 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568755+0000", 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9300000000000006e-07 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568847+0000", 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1906999999999999e-05 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568859+0000", 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2047e-05 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568866+0000", 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7170000000000002e-06 2024-09-15T02:49:15.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580440+0000", 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011574442000000001 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580500+0000", 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9732999999999999e-05 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.580611+0000", 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011137999999999999 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583173+0000", 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025617330000000001 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586614+0000", 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034406929999999999 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586671+0000", 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7398000000000002e-05 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586708+0000", 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7024000000000003e-05 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:15.933 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.304704073000003, 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:15.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:15.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.287185497000003, 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:15.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:15.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 48.473323069999999, 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:15.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:15.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.5s3 41/33 ECSubWrite(tid=20, reqid=client.4526.0:249, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:45.591250+0000", 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.282327504000001, 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017544212999999999, 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591250+0000", 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591250+0000", 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591250+0000", 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7700000000000001e-07 2024-09-15T02:49:15.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591716+0000", 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046586199999999999 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591752+0000", 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5995999999999999e-05 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591760+0000", 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1040000000000007e-06 2024-09-15T02:49:15.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591811+0000", 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0735999999999997e-05 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.591820+0000", 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1849999999999993e-06 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.608760+0000", 2024-09-15T02:49:15.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016940068999999999 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:45.608794+0000", 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3983999999999999e-05 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:312 3.2s0 3:48b768c1:::smithi19125844-48:head [append 0~802816] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.648236+0000", 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.225341847999999, 2024-09-15T02:49:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018776604999999998, 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 312 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648236+0000", 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648236+0000", 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648238+0000", 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.905e-06 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648557+0000", 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031915300000000003 2024-09-15T02:49:15.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648579+0000", 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1733999999999999e-05 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648587+0000", 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5839999999999995e-06 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648666+0000", 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8652000000000005e-05 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.648812+0000", 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000146379 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650328+0000", 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015154739999999999 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.663451+0000", 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013123212 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666936+0000", 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034846909999999998 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.667012+0000", 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6821000000000002e-05 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.167079432000001, 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.962 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:15.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:15.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.167021599000002, 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:15.968 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:15.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.973 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.166986413, 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:15.974 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:15.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.es1 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.931374+0000", 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.942203227, 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020869234, 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.979 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931374+0000", 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931374+0000", 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931375+0000", 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0900000000000001e-07 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931828+0000", 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045379699999999998 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931962+0000", 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013329999999999999 2024-09-15T02:49:15.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931969+0000", 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2609999999999996e-06 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932088+0000", 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118706 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932107+0000", 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9185999999999999e-05 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.952226+0000", 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020119103999999999 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.952244+0000", 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7671000000000001e-05 2024-09-15T02:49:15.983 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.431681854000001, 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:15.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.988 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.593249641, 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:15.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:555 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~491520] snapc 18=[18,15,14,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:05.351404+0000", 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.522173348000001, 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017949692999999999, 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:15.992 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 555 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351404+0000", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351404+0000", 2024-09-15T02:49:15.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351407+0000", 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.181e-06 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351866+0000", 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045943200000000002 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:15.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351878+0000", 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.226e-05 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351886+0000", 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2800000000000003e-06 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.351951+0000", 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4981999999999995e-05 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.352091+0000", 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000139128 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.353148+0000", 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001057445 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.363179+0000", 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010031201 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.369275+0000", 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060960809999999997 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:05.369354+0000", 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8702999999999995e-05 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:15.997 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:15.999 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:15.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:15.923+0000 7f6402ffd700 0 -- 172.21.15.6:0/1962617802 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f6404192f80 msgr2=0x7f6404195360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:16.000 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:16.000 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:16.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:16.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:16.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:16.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:16.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:16.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:16.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:16.003 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:15.992+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:16.100 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:16.092+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:16.110 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:16.110 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:16.110 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:16.110 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:16.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:16.201 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:16.196+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:16.242 INFO:tasks.rados.rados.0.smithi191.stdout:256: done (4 left) 2024-09-15T02:49:16.242 INFO:tasks.rados.rados.0.smithi191.stdout:258: done (3 left) 2024-09-15T02:49:16.242 INFO:tasks.rados.rados.0.smithi191.stdout:260: done (2 left) 2024-09-15T02:49:16.242 INFO:tasks.rados.rados.0.smithi191.stdout:261: done (1 left) 2024-09-15T02:49:16.242 INFO:tasks.rados.rados.0.smithi191.stdout:263: done (0 left) 2024-09-15T02:49:16.243 INFO:tasks.rados.rados.0.smithi191.stdout:264: delete oid 6 current snap is 20 2024-09-15T02:49:16.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:49:16.261 INFO:tasks.rados.rados.0.smithi191.stdout:264: done (0 left) 2024-09-15T02:49:16.261 INFO:tasks.rados.rados.0.smithi191.stdout:265: read oid 44 snap -1 2024-09-15T02:49:16.261 INFO:tasks.rados.rados.0.smithi191.stdout:265: expect (ObjNum 98 snap 15 seq_num 98) 2024-09-15T02:49:16.261 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 43 current snap is 20 2024-09-15T02:49:16.261 INFO:tasks.rados.rados.0.smithi191.stdout:266: seq_num 109 ranges {0=671744,671744=753664,1425408=409600} 2024-09-15T02:49:16.270 INFO:tasks.rados.rados.0.smithi191.stdout:266: writing smithi19125844-43 from 0 to 671744 tid 1 2024-09-15T02:49:16.276 INFO:tasks.rados.rados.0.smithi191.stdout:266: writing smithi19125844-43 from 671744 to 1425408 tid 2 2024-09-15T02:49:16.280 INFO:tasks.rados.rados.0.smithi191.stdout:266: writing smithi19125844-43 from 1425408 to 1835008 tid 3 2024-09-15T02:49:16.280 INFO:tasks.rados.rados.0.smithi191.stdout:267: read oid 31 snap -1 2024-09-15T02:49:16.280 INFO:tasks.rados.rados.0.smithi191.stdout:267: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-15T02:49:16.280 INFO:tasks.rados.rados.0.smithi191.stdout:268: read oid 5 snap 14 2024-09-15T02:49:16.280 INFO:tasks.rados.rados.0.smithi191.stdout:268: expect (ObjNum 68 snap 5 seq_num 68) 2024-09-15T02:49:16.280 INFO:tasks.rados.rados.0.smithi191.stdout:269: delete oid 26 current snap is 20 2024-09-15T02:49:16.295 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:16.295 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:16.295 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:16.296 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:16.296 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:16.301 INFO:tasks.rados.rados.0.smithi191.stdout:267: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-15T02:49:16.302 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:16.296+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:16.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:16.318 INFO:tasks.rados.rados.0.smithi191.stdout:268: expect (ObjNum 68 snap 5 seq_num 68) 2024-09-15T02:49:16.403 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:16.396+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:16.418 INFO:tasks.rados.rados.0.smithi191.stdout:266: finishing write tid 1 to smithi19125844-43 2024-09-15T02:49:16.418 INFO:tasks.rados.rados.0.smithi191.stdout:266: finishing write tid 2 to smithi19125844-43 2024-09-15T02:49:16.418 INFO:tasks.rados.rados.0.smithi191.stdout:266: finishing write tid 3 to smithi19125844-43 2024-09-15T02:49:16.419 INFO:tasks.rados.rados.0.smithi191.stdout:267: done (4 left) 2024-09-15T02:49:16.419 INFO:tasks.rados.rados.0.smithi191.stdout:268: done (3 left) 2024-09-15T02:49:16.419 INFO:tasks.rados.rados.0.smithi191.stdout:269: done (2 left) 2024-09-15T02:49:16.419 INFO:tasks.rados.rados.0.smithi191.stdout:270: read oid 21 snap -1 2024-09-15T02:49:16.419 INFO:tasks.rados.rados.0.smithi191.stdout:270: expect (ObjNum 106 snap 19 seq_num 106) 2024-09-15T02:49:16.419 INFO:tasks.rados.rados.0.smithi191.stdout:271: snap_create 2024-09-15T02:49:16.420 INFO:tasks.rados.rados.0.smithi191.stdout:266: finishing write tid 4 to smithi19125844-43 2024-09-15T02:49:16.420 INFO:tasks.rados.rados.0.smithi191.stdout:265: expect (ObjNum 98 snap 15 seq_num 98) 2024-09-15T02:49:16.422 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:16.422 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:16.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:16.422 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:16.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/33 ECSubWriteReply(tid=19, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.875583915, 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74030938599999996, 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:16.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539611+0000", 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3680000000000001e-06 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539619+0000", 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278674+0000", 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73905563200000002 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278687+0000", 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.239e-05 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278784+0000", 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7149000000000001e-05 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.279919+0000", 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001135645 2024-09-15T02:49:16.426 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:234 3.3s0 3:cebed992:::smithi19125844-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.565967+0000", 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 32.849227022999997, 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012231558, 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.427 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 234 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565967+0000", 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565967+0000", 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565969+0000", 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.891e-06 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565976+0000", 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0570000000000004e-06 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.565997+0000", 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0678000000000001e-05 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566005+0000", 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2349999999999999e-06 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566086+0000", 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1392999999999998e-05 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566098+0000", 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1361e-05 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570199+0000", 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041015019999999999 2024-09-15T02:49:16.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570317+0000", 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117787 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.571897+0000", 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001580674 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577044+0000", 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051465269999999997 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578083+0000", 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001038612 2024-09-15T02:49:16.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.578198+0000", 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115841 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:235 3.3s0 3:cebed992:::smithi19125844-27:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.566032+0000", 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 32.849161586000001, 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014205251, 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 235 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566032+0000", 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566032+0000", 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566033+0000", 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6300000000000005e-07 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566048+0000", 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5051999999999999e-05 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566056+0000", 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566066+0000", 2024-09-15T02:49:16.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8630000000000003e-06 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566224+0000", 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000157637 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566250+0000", 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6570000000000001e-05 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572420+0000", 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006169557 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.572522+0000", 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101848 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.573712+0000", 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011904319999999999 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577493+0000", 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037809860000000001 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580068+0000", 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025749179999999998 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.580238+0000", 2024-09-15T02:49:16.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016964899999999999 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:236 3.3s0 3:cebed992:::smithi19125844-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 32.849121402999998, 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018295445, 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 236 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:49:16.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566073+0000", 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1600000000000001e-07 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566075+0000", 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2359999999999999e-06 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566083+0000", 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1410000000000005e-06 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566091+0000", 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8529999999999992e-06 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566291+0000", 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000199242 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.566295+0000", 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.972e-06 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574064+0000", 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077684240000000003 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574167+0000", 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010275 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.574540+0000", 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037325999999999999 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.577496+0000", 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002956242 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584322+0000", 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0068258470000000003 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.584368+0000", 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5961999999999998e-05 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=24, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 32.828216626, 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30338358999999998, 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.449 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586978+0000", 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.51e-07 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586981+0000", 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890125+0000", 2024-09-15T02:49:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30314395700000002 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890138+0000", 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2627e-05 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890297+0000", 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015958899999999999 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890361+0000", 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3658999999999995e-05 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:284 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:16.453 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598091+0000", 2024-09-15T02:49:16.454 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.817102837, 2024-09-15T02:49:16.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019191204999999999, 2024-09-15T02:49:16.454 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.454 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:16.454 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.454 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 284 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598091+0000", 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598091+0000", 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598095+0000", 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8199999999999998e-06 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598114+0000", 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9202999999999999e-05 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598163+0000", 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8420000000000001e-05 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598190+0000", 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7555999999999998e-05 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598409+0000", 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021893499999999999 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598431+0000", 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1843000000000001e-05 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612108+0000", 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013677116 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612283+0000", 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000175107 2024-09-15T02:49:16.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617282+0000", 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0049992049999999996 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:285 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.816922762000001, 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019260777999999999, 2024-09-15T02:49:16.460 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 285 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:16.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598273+0000", 2024-09-15T02:49:16.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1670000000000002e-06 2024-09-15T02:49:16.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598285+0000", 2024-09-15T02:49:16.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1873e-05 2024-09-15T02:49:16.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598308+0000", 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2895999999999999e-05 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598321+0000", 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3293e-05 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598586+0000", 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026496499999999998 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598606+0000", 2024-09-15T02:49:16.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9760000000000001e-05 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613160+0000", 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014553452 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613251+0000", 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1471000000000006e-05 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617532+0000", 2024-09-15T02:49:16.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042809010000000002 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:286 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.816759734000001, 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019362717000000002, 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 286 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598437+0000", 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4940000000000002e-06 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598451+0000", 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.448e-05 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598503+0000", 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1983e-05 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598527+0000", 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4026e-05 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598660+0000", 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013317900000000001 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598681+0000", 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0296e-05 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613559+0000", 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014878416 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613646+0000", 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6667000000000002e-05 2024-09-15T02:49:16.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617797+0000", 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041511760000000003 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:414 3.3s0 3:cebed992:::smithi19125844-27:head [copy-from ver 4] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.159330191999999, 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034868557000000001, 2024-09-15T02:49:16.473 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 414 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255866+0000", 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.621e-06 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255879+0000", 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2738999999999999e-05 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255912+0000", 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3099000000000003e-05 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255927+0000", 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5013e-05 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256006+0000", 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8688000000000003e-05 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256016+0000", 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8379999999999997e-06 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262150+0000", 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061347040000000004 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262292+0000", 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141595 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.275991+0000", 2024-09-15T02:49:16.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013699266999999999 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282717+0000", 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067257910000000001 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290669+0000", 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079519480000000003 2024-09-15T02:49:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290732+0000", 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3254000000000001e-05 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:415 3.3s0 3:cebed992:::smithi19125844-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.159242573, 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026655980999999999, 2024-09-15T02:49:16.480 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 415 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255952+0000", 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.108e-06 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255956+0000", 2024-09-15T02:49:16.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.597e-06 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255965+0000", 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3789999999999995e-06 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255971+0000", 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3199999999999999e-06 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256090+0000", 2024-09-15T02:49:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011949 2024-09-15T02:49:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256109+0000", 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8644999999999999e-05 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262493+0000", 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063841519999999997 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262538+0000", 2024-09-15T02:49:16.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4790999999999999e-05 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282336+0000", 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019798651 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282486+0000", 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149037 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282607+0000", 2024-09-15T02:49:16.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121811 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=77, last_complete=68'67, committed=1, applied=1))", 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.0028331509999999, 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.106159008, 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412362+0000", 2024-09-15T02:49:16.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9999999999999996e-07 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412370+0000", 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4120000000000001e-06 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517947+0000", 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10557704800000001 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517957+0000", 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0173999999999999e-05 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518475+0000", 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051777099999999999 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518520+0000", 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4703000000000002e-05 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.490 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:636 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~606208] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.918178004, 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026984608, 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 636 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497019+0000", 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.066e-06 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497546+0000", 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052673800000000001 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497580+0000", 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4647999999999998e-05 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497592+0000", 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1669e-05 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497695+0000", 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103031 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497915+0000", 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219607 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.499984+0000", 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020692800000000002 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518517+0000", 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018533353999999998 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523868+0000", 2024-09-15T02:49:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053511779999999998 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524001+0000", 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132037 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:637 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~688128] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.9101207819999999, 2024-09-15T02:49:16.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022414014999999999, 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 637 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505076+0000", 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0570000000000001e-06 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505732+0000", 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065579899999999999 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505759+0000", 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6744999999999999e-05 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505786+0000", 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7518999999999999e-05 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.506354+0000", 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00056810900000000002 2024-09-15T02:49:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.506548+0000", 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000193263 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508427+0000", 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018793289999999999 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523607+0000", 2024-09-15T02:49:16.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015180023000000001 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527407+0000", 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038003220000000001 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527487+0000", 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9849000000000001e-05 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:638 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~475136] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.907078405, 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.170696291, 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 638 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508118+0000", 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6110000000000001e-06 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508908+0000", 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00078945100000000004 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508947+0000", 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9298999999999998e-05 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508971+0000", 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4329e-05 2024-09-15T02:49:16.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518663+0000", 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0096915070000000002 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518760+0000", 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7740999999999998e-05 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.519373+0000", 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061209299999999997 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523608+0000", 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042356319999999996 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678746+0000", 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15513796099999999 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678812+0000", 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5667000000000007e-05 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:639 3.3s0 3:c4165a1c:::smithi19125844-22:head [setxattr _header (57) in=7b] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.9061938820000002, 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.189219685, 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:16.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 639 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509002+0000", 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7579999999999999e-06 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509010+0000", 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3110000000000006e-06 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509031+0000", 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1234e-05 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509052+0000", 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0962e-05 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522387+0000", 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013334693 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522514+0000", 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126723 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523057+0000", 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054339999999999998 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524433+0000", 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001375739 2024-09-15T02:49:16.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698072+0000", 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17363879099999999 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698220+0000", 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014807399999999999 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:640 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.906106651, 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19189399200000001, 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 640 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509090+0000", 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2800000000000002e-06 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509095+0000", 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9679999999999997e-06 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509113+0000", 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8031000000000001e-05 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509122+0000", 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6509999999999996e-06 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524120+0000", 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014997610999999999 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524399+0000", 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027887000000000002 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698989+0000", 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.174589943 2024-09-15T02:49:16.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.699138+0000", 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149333 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.700981+0000", 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018433049999999999 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=80, last_complete=68'71, committed=1, applied=1))", 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.8879801230000002, 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15161024100000001, 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527215+0000", 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.088e-06 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527221+0000", 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9909999999999999e-06 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678531+0000", 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15131061000000001 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678545+0000", 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3597e-05 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678666+0000", 2024-09-15T02:49:16.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012109599999999999 2024-09-15T02:49:16.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678824+0000", 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015785899999999999 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=81, last_complete=68'72, committed=1, applied=1))", 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.7367176660000001, 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020448245, 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678478+0000", 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6729999999999999e-06 2024-09-15T02:49:16.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678489+0000", 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.059e-05 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697765+0000", 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0192768 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697777+0000", 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1749000000000001e-05 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697916+0000", 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013848199999999999 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698925+0000", 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010089509999999999 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.525 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs3 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.274372+0000", 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.14082208500000001, 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018000322999999999, 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274372+0000", 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274372+0000", 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274372+0000", 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.01e-07 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274647+0000", 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027514500000000002 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274687+0000", 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9365e-05 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274696+0000", 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274752+0000", 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5883e-05 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.274763+0000", 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1182000000000001e-05 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292340+0000", 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017577133000000002 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292372+0000", 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2274000000000003e-05 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs3 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.284325+0000", 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.13086889900000001, 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018933905000000001, 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:16.530 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284325+0000", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284325+0000", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284345+0000", 2024-09-15T02:49:16.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9777999999999999e-05 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284564+0000", 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021885599999999999 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284603+0000", 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9582000000000002e-05 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284610+0000", 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2320000000000004e-06 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284876+0000", 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026514799999999999 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284882+0000", 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1099999999999999e-06 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.303210+0000", 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018327766999999998 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.303259+0000", 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9431999999999997e-05 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:16.534 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:16.536 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:16.536 INFO:tasks.rados.rados.0.smithi191.stdout:266: finishing write tid 5 to smithi19125844-43 2024-09-15T02:49:16.537 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 81 (ObjNum 109 snap 20 seq_num 109) dirty exists 2024-09-15T02:49:16.537 INFO:tasks.rados.rados.0.smithi191.stdout:266: left oid 43 (ObjNum 109 snap 20 seq_num 109) 2024-09-15T02:49:16.537 INFO:tasks.rados.rados.0.smithi191.stdout:270: expect (ObjNum 106 snap 19 seq_num 106) 2024-09-15T02:49:16.537 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:16.528+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:16.636 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:16.634+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:16.646 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:16.646 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:16.646 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:16.646 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:16.655 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:16.737 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:16.732+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:16.758 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:16.758 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:16.758 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:16.759 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:16.759 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:16.768 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:16.838 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:16.836+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:871775f5:::smithi19125844-15:c, version: 48'19, data_included: [0~61440], data_size: 61440, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:871775f5:::smithi19125844-15:c@48'19, size: 245760, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:245760, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.420810159999998, 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80849678800000002, 2024-09-15T02:49:16.868 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442896+0000", 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0449999999999999e-06 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442986+0000", 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9937000000000003e-05 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.246998+0000", 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80401223499999996 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247018+0000", 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9735999999999999e-05 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247164+0000", 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000146093 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.251392+0000", 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042277419999999996 2024-09-15T02:49:16.871 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:801a1da7:::smithi19125844-39:e, version: 49'21, data_included: [0~159744], data_size: 159744, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:e@49'21, size: 638976, copy_subset: [], clone_subset: {}, snapset: e=[]:{e=[e,b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:638976, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.598080106000001, 2024-09-15T02:49:16.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010047222999999999, 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265626+0000", 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.325e-06 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265815+0000", 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018882999999999999 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265866+0000", 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0565000000000001e-05 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265876+0000", 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0273e-05 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.266085+0000", 2024-09-15T02:49:16.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000209049 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.275672+0000", 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095871810000000002 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.579482142, 2024-09-15T02:49:16.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087482150000000002, 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284497+0000", 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027328299999999998 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284538+0000", 2024-09-15T02:49:16.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1449999999999998e-05 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284547+0000", 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1020000000000003e-06 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284639+0000", 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1615999999999998e-05 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.292971+0000", 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083324509999999994 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [262144~221184], data_size: 221184, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1933312, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.575166769999999, 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014791191, 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.881 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288540+0000", 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9369999999999998e-06 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288962+0000", 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042141100000000001 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289058+0000", 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6339e-05 2024-09-15T02:49:16.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289089+0000", 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0867000000000001e-05 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289255+0000", 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016596199999999999 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.303330+0000", 2024-09-15T02:49:16.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014074675 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:e, version: 49'26, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:e@49'26, size: 1081344, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.554702991999999, 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007188888, 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309004+0000", 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4529999999999999e-06 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309408+0000", 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000404798 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309488+0000", 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9389999999999997e-05 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309520+0000", 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2509999999999999e-05 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309641+0000", 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120958 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.316191+0000", 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0065497790000000004 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.888 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.889 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.889 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.889 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:16.889 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.538603116000001, 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012696989000000001, 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325103+0000", 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.333e-06 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325526+0000", 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000422788 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325605+0000", 2024-09-15T02:49:16.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8932999999999998e-05 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325624+0000", 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8525999999999998e-05 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325780+0000", 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015595999999999999 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.337799+0000", 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012019449 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.895 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.895 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:16.895 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.521238025999999, 2024-09-15T02:49:16.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01280455, 2024-09-15T02:49:16.895 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.895 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.896 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:16.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342469+0000", 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6929999999999999e-06 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342850+0000", 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000381462 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342951+0000", 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000100559 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342971+0000", 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9720000000000001e-05 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.343066+0000", 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5088999999999997e-05 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.355272+0000", 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012206027 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.493543936000002, 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012408221000000001, 2024-09-15T02:49:16.900 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370163+0000", 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370553+0000", 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038988100000000002 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370636+0000", 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2825999999999996e-05 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370655+0000", 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.916e-05 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370778+0000", 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012348700000000001 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.382570+0000", 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01179122 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.477428153000002, 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070382149999999996, 2024-09-15T02:49:16.904 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.905 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.905 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:16.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:16.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386278+0000", 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2699999999999999e-06 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386706+0000", 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042724800000000003 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386783+0000", 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6976000000000003e-05 2024-09-15T02:49:16.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386809+0000", 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6143e-05 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386892+0000", 2024-09-15T02:49:16.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3196999999999996e-05 2024-09-15T02:49:16.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.393315+0000", 2024-09-15T02:49:16.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0064233809999999997 2024-09-15T02:49:16.911 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.911 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.911 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.911 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.912 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:16.912 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.466451882000001, 2024-09-15T02:49:16.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0076400929999999997, 2024-09-15T02:49:16.912 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.912 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.912 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397255+0000", 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.576e-06 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397632+0000", 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000377265 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397715+0000", 2024-09-15T02:49:16.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3213000000000004e-05 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397743+0000", 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7229999999999998e-05 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397885+0000", 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014198699999999999 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.404893+0000", 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070088219999999996 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [786432~167936], data_size: 167936, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3817472, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.454689372000001, 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053001360000000004, 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.916 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409017+0000", 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.406e-06 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409301+0000", 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028334600000000002 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409374+0000", 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3652999999999994e-05 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409421+0000", 2024-09-15T02:49:16.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6879000000000003e-05 2024-09-15T02:49:16.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409504+0000", 2024-09-15T02:49:16.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2559999999999996e-05 2024-09-15T02:49:16.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.414316+0000", 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048122920000000001 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.921 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/53 ECSubWrite(tid=67, reqid=client.4526.0:522, at_version=60'31, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.596157518, 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43332960599999998, 2024-09-15T02:49:16.922 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:16.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8200000000000001e-07 2024-09-15T02:49:16.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.925 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.925 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267569+0000", 2024-09-15T02:49:16.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0713e-05 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699630+0000", 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43206165099999999 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699644+0000", 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3274e-05 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699794+0000", 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000150719 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699816+0000", 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1895999999999998e-05 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700821+0000", 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010047210000000001 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700877+0000", 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6249999999999998e-05 2024-09-15T02:49:16.928 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 61/53 ECSubWrite(tid=102, reqid=osd.14.0:99, at_version=0'0, trim_to=0'0, roll_forward_to=61'43))", 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.514408506000001, 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0054209050000000002, 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7499999999999999e-07 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349311+0000", 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3689999999999999e-05 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349385+0000", 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3937999999999996e-05 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349392+0000", 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0790000000000003e-06 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349455+0000", 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2742000000000003e-05 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349465+0000", 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0259e-05 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354685+0000", 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052197889999999999 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354718+0000", 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2632999999999997e-05 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 68/53 ECSubWrite(tid=123, reqid=client.4526.0:630, at_version=68'45, trim_to=0'0, roll_forward_to=61'43))", 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.409801+0000", 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.4539038590000004, 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0047233930000000002, 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409801+0000", 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409801+0000", 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409802+0000", 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1699999999999999e-07 2024-09-15T02:49:16.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409847+0000", 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5602999999999997e-05 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409879+0000", 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2002000000000003e-05 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409891+0000", 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2109999999999999e-05 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409981+0000", 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9287000000000001e-05 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409997+0000", 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6248000000000001e-05 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414485+0000", 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004488344 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414525+0000", 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9481999999999999e-05 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=402, reqid=osd.13.0:402, at_version=72'63, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:16.938 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.258650+0000", 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.605054867, 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060995349999999997, 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258650+0000", 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258650+0000", 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258651+0000", 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.84e-07 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258674+0000", 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2929999999999999e-05 2024-09-15T02:49:16.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258779+0000", 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010520500000000001 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258789+0000", 2024-09-15T02:49:16.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0504e-05 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258848+0000", 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8316000000000002e-05 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258864+0000", 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6021000000000001e-05 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264720+0000", 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058567489999999996 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264750+0000", 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9526000000000002e-05 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.945 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=403, reqid=osd.13.0:403, at_version=72'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.60502908, 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0085463759999999996, 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.946 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8500000000000002e-07 2024-09-15T02:49:16.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258694+0000", 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7482e-05 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258812+0000", 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000118245 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258817+0000", 2024-09-15T02:49:16.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4379999999999997e-06 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259246+0000", 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042895599999999998 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259256+0000", 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0818e-05 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.267192+0000", 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0079355799999999994 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.267223+0000", 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0571999999999998e-05 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 72/53 ECSubWrite(tid=96, reqid=client.4526.0:667, at_version=0'0, trim_to=0'0, roll_forward_to=72'53))", 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.259587+0000", 2024-09-15T02:49:16.950 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.60411781499999995, 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052448989999999999, 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259587+0000", 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259587+0000", 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259588+0000", 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0759999999999999e-06 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259602+0000", 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3128e-05 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259656+0000", 2024-09-15T02:49:16.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4800999999999999e-05 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259665+0000", 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6999999999999997e-06 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259750+0000", 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4556999999999997e-05 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259761+0000", 2024-09-15T02:49:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1652e-05 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264807+0000", 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050455769999999999 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264832+0000", 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5408000000000001e-05 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.19s6 72/71 ECSubWrite(tid=132, reqid=osd.14.0:132, at_version=72'51, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.58798070999999996, 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0057855809999999997, 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9899999999999999e-07 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275754+0000", 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8892999999999999e-05 2024-09-15T02:49:16.956 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275812+0000", 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8069000000000003e-05 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275821+0000", 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5420000000000005e-06 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275876+0000", 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5547999999999997e-05 2024-09-15T02:49:16.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275893+0000", 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6863000000000001e-05 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.281483+0000", 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055895529999999997 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.281510+0000", 2024-09-15T02:49:16.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7214000000000001e-05 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=409, reqid=osd.13.0:409, at_version=72'67, trim_to=0'0, roll_forward_to=72'65))", 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.57371039499999998, 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050522639999999999, 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.959 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4200000000000002e-07 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.960 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290019+0000", 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3717e-05 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290089+0000", 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0205999999999998e-05 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290095+0000", 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9560000000000002e-06 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290171+0000", 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6384000000000006e-05 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290185+0000", 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3219e-05 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.295015+0000", 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048300770000000003 2024-09-15T02:49:16.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.295047+0000", 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2462999999999997e-05 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.19s6 72/71 ECSubWrite(tid=134, reqid=osd.14.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=72'51))", 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.291564+0000", 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.57214077799999996, 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.006517892, 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:16.963 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291564+0000", 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291564+0000", 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:16.964 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291566+0000", 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.215e-06 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291583+0000", 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7119e-05 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291640+0000", 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7188000000000002e-05 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291651+0000", 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1365e-05 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291705+0000", 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3922000000000003e-05 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291716+0000", 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0862999999999999e-05 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.298052+0000", 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0063362660000000001 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.298082+0000", 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9954000000000002e-05 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:16.967 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:16.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:16.969 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:16.964+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:17.070 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:17.063+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:17.079 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:17.079 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:17.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:17.175 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:17.164+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:17.235 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:17.235 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:17.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:17.272 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:17.268+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 26.696746591, 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01495401, 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649416+0000", 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649418+0000", 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.553e-06 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649682+0000", 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026439200000000001 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649749+0000", 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6585e-05 2024-09-15T02:49:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649761+0000", 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1584e-05 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649838+0000", 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7866000000000003e-05 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649852+0000", 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4e-05 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664337+0000", 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014485031000000001 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664370+0000", 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2999e-05 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.586935958000002, 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015371035, 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759227+0000", 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759228+0000", 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3599999999999999e-06 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759805+0000", 2024-09-15T02:49:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057638899999999996 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759906+0000", 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010075499999999999 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.759920+0000", 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4166e-05 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.760017+0000", 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7647999999999997e-05 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.760038+0000", 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0290999999999999e-05 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.774535+0000", 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014497252 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.774598+0000", 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3174000000000002e-05 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:422 3.cs0 3:319e3fcc:::smithi19125844-50:head [rollback a] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.079693478999999, 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028855308, 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 422 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266470+0000", 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.85e-07 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266475+0000", 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.527e-06 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266498+0000", 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3652999999999999e-05 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266521+0000", 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3006999999999998e-05 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268383+0000", 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018616730000000001 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268454+0000", 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0479000000000001e-05 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275682+0000", 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072287380000000002 2024-09-15T02:49:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.283977+0000", 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082944760000000003 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295223+0000", 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011245890999999999 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295325+0000", 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102079 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:423 3.cs0 3:319e3fcc:::smithi19125844-50:head [append 0~0] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.079623521999999, 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028955272000000001, 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 423 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266539+0000", 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266541+0000", 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5719999999999999e-06 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266548+0000", 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4899999999999997e-06 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266559+0000", 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1158000000000001e-05 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.266564+0000", 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1630000000000004e-06 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276125+0000", 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095610759999999999 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276243+0000", 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117991 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276700+0000", 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00045704300000000002 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286454+0000", 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097542040000000007 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295420+0000", 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089656649999999994 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.295495+0000", 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4909999999999999e-05 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.071063145, 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014524752, 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275100+0000", 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275101+0000", 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6899999999999996e-07 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275587+0000", 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000485976 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275641+0000", 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4728999999999997e-05 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275650+0000", 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4749999999999993e-06 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275756+0000", 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105877 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275768+0000", 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2734999999999999e-05 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289496+0000", 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013727892 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289625+0000", 2024-09-15T02:49:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012829900000000001 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:512 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~786432] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.102796529000001, 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026925933999999999, 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 512 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243366+0000", 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.243369+0000", 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7709999999999998e-06 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.245943+0000", 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025742260000000002 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.245980+0000", 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6510000000000001e-05 2024-09-15T02:49:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246008+0000", 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8286e-05 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246085+0000", 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6760000000000004e-05 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246181+0000", 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5546999999999998e-05 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.247292+0000", 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011118090000000001 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261781+0000", 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014488861 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270200+0000", 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084192450000000005 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270292+0000", 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1918999999999999e-05 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:513 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~491520] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.099876836, 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024200269, 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 513 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246286+0000", 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.246289+0000", 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7329999999999999e-06 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248021+0000", 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017319410000000001 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248044+0000", 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3451999999999998e-05 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248057+0000", 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2745000000000001e-05 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.253501+0000", 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054438999999999998 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.253636+0000", 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000134587 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.254328+0000", 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000692474 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074557210000000002 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270429+0000", 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0086448540000000004 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.270486+0000", 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7862000000000003e-05 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:514 3.fs0 3:f0e056ea:::smithi19125844-7:head [append 0~278528] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.098035544, 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023758102, 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 514 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248127+0000", 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248129+0000", 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4389999999999999e-06 2024-09-15T02:49:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248897+0000", 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000768591 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248919+0000", 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1985999999999999e-05 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248931+0000", 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1516e-05 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.257681+0000", 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087502150000000004 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.257788+0000", 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106774 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.258392+0000", 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060445900000000001 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266310+0000", 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079177050000000006 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271830+0000", 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055203140000000001 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271886+0000", 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5102999999999997e-05 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:515 3.fs0 3:f0e056ea:::smithi19125844-7:head [setxattr _header (56) in=7b] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.097222621, 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023210873999999999, 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 515 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248940+0000", 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248941+0000", 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.035e-06 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248945+0000", 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0330000000000002e-06 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248957+0000", 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1669e-05 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248962+0000", 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2650000000000004e-06 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.260634+0000", 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011671488000000001 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.260737+0000", 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102816 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261347+0000", 2024-09-15T02:49:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000610043 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266315+0000", 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049678099999999996 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272042+0000", 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005727869 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272151+0000", 2024-09-15T02:49:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010884599999999999 2024-09-15T02:49:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:516 3.fs0 3:f0e056ea:::smithi19125844-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.097194993, 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.025266506000000001, 2024-09-15T02:49:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 516 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248968+0000", 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248969+0000", 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4700000000000001e-07 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248971+0000", 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.588e-06 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248983+0000", 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1579999999999999e-05 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.248995+0000", 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.185e-05 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261706+0000", 2024-09-15T02:49:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012710971 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261774+0000", 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8342000000000004e-05 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272225+0000", 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010450971999999999 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.272297+0000", 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1903999999999995e-05 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.274235+0000", 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019377520000000001 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:520 3.fs0 3:f39d968a:::smithi19125844-6:head [append 0~0] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.097163126, 2024-09-15T02:49:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024220364000000001, 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 520 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249000+0000", 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2100000000000002e-07 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249003+0000", 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4779999999999998e-06 2024-09-15T02:49:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249010+0000", 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1799999999999999e-06 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249018+0000", 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1300000000000001e-06 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261843+0000", 2024-09-15T02:49:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012824894999999999 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262019+0000", 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017557599999999999 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263083+0000", 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001064206 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266322+0000", 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032392810000000001 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273128+0000", 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068061969999999999 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273220+0000", 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2e-05 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:521 3.fs0 3:f39d968a:::smithi19125844-6:head [rollback 15] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.097139196000001, 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024414946999999999, 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 521 2024-09-15T02:49:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249024+0000", 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3700000000000001e-07 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249027+0000", 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7819999999999998e-06 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249038+0000", 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0569e-05 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249043+0000", 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2129999999999999e-06 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263801+0000", 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014758603 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263989+0000", 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000188004 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.264897+0000", 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090739499999999997 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.268454+0000", 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035574119999999998 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273370+0000", 2024-09-15T02:49:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049154689999999996 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.273439+0000", 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9062999999999999e-05 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:522 3.fs0 3:f39d968a:::smithi19125844-6:head [append 0~0] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.097108885000001, 2024-09-15T02:49:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45227964500000001, 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 522 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249054+0000", 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249055+0000", 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0800000000000005e-07 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249058+0000", 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.914e-06 2024-09-15T02:49:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249068+0000", 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0883000000000001e-05 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.249073+0000", 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5399999999999997e-06 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.265365+0000", 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016292469 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.265485+0000", 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011935 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.266071+0000", 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058648000000000001 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.271231+0000", 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051601140000000004 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.701218+0000", 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.42998620999999998 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.701334+0000", 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116177 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 60/33 ECSubWrite(tid=75, reqid=client.4526.0:529, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.084700707, 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022861161000000001, 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261462+0000", 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261463+0000", 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.13e-06 2024-09-15T02:49:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261848+0000", 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038425800000000001 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261906+0000", 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7896000000000003e-05 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261913+0000", 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8269999999999998e-06 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261989+0000", 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.559e-05 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262002+0000", 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3291e-05 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284285+0000", 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022282659999999999 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.284323+0000", 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8509e-05 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=64, reqid=client.4526.0:610, at_version=66'54, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.510547+0000", 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.8356161479999997, 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01556914, 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510547+0000", 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510547+0000", 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510548+0000", 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.643e-06 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511151+0000", 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060209599999999999 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511226+0000", 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5841999999999997e-05 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511238+0000", 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1731e-05 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511301+0000", 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2636000000000001e-05 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.511316+0000", 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.472e-05 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.525996+0000", 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014680901 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526116+0000", 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119571 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 66/33 ECSubWrite(tid=65, reqid=client.4526.0:611, at_version=66'55, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.518894+0000", 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.8272693110000002, 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019660654, 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518894+0000", 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518894+0000", 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.518895+0000", 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8299999999999995e-07 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519184+0000", 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00028936100000000002 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519222+0000", 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7929999999999998e-05 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519231+0000", 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0399999999999998e-06 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519369+0000", 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013751400000000001 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.519435+0000", 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6125999999999997e-05 2024-09-15T02:49:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.538522+0000", 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019087197 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.538554+0000", 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2502999999999997e-05 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 68/33 ECSubWrite(tid=78, reqid=client.4526.0:636, at_version=68'69, trim_to=0'0, roll_forward_to=68'67))", 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:12.499300+0000", 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.8468630810000004, 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.018078766999999999, 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499300+0000", 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499300+0000", 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499301+0000", 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.497e-06 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499817+0000", 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051576699999999998 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499956+0000", 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013931499999999999 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499969+0000", 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3006999999999999e-05 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500198+0000", 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022864900000000001 2024-09-15T02:49:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500231+0000", 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3113e-05 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.517337+0000", 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017105829999999999 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.517379+0000", 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1588999999999998e-05 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 71/53 ECSubWriteReply(tid=92, last_complete=71'51, committed=1, applied=1))", 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:15.256817+0000", 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 2.0893461279999999, 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.936271293, 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:15.256817+0000", 2024-09-15T02:49:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:15.256817+0000", 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:15.256817+0000", 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:15.256819+0000", 2024-09-15T02:49:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.657e-06 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.191370+0000", 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.93455105500000002 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.191384+0000", 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4365999999999999e-05 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.191530+0000", 2024-09-15T02:49:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000146033 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.193088+0000", 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001557987 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:667 3.fs0 3:f39d968a:::smithi19125844-6:head [stat,delete] snapc 24=[24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:16.241315+0000", 2024-09-15T02:49:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.1048482550000001, 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017820177999999999, 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 667 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241315+0000", 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241315+0000", 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241317+0000", 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7999999999999999e-06 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241324+0000", 2024-09-15T02:49:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4250000000000001e-06 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241346+0000", 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2504e-05 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241361+0000", 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4541e-05 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241435+0000", 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4179999999999998e-05 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.241443+0000", 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1890000000000007e-06 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.247344+0000", 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059002220000000001 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.247603+0000", 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025984800000000002 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.248550+0000", 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00094649700000000003 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.250180+0000", 2024-09-15T02:49:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001629697 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.259020+0000", 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0088399559999999995 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.259135+0000", 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115319 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:682 3.fs0 3:fc5a0702:::smithi19125844-26:head [stat,delete] snapc 24=[24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:16.279112+0000", 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.0670510120000001, 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014350843, 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 682 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.279112+0000", 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.279112+0000", 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.279113+0000", 2024-09-15T02:49:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.44e-07 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.279115+0000", 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8210000000000002e-06 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.279123+0000", 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1930000000000004e-06 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.279127+0000", 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2849999999999996e-06 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.281581+0000", 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024538329999999999 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.281754+0000", 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017344300000000001 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.282597+0000", 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00084322300000000004 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.286019+0000", 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034221120000000002 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.293355+0000", 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073351019999999996 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.293463+0000", 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000108287 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.456 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:17.457 INFO:tasks.rados.rados.0.smithi191.stdout:265: done (3 left) 2024-09-15T02:49:17.457 INFO:tasks.rados.rados.0.smithi191.stdout:266: done (2 left) 2024-09-15T02:49:17.457 INFO:tasks.rados.rados.0.smithi191.stdout:270: done (1 left) 2024-09-15T02:49:17.457 INFO:tasks.rados.rados.0.smithi191.stdout:271: done (0 left) 2024-09-15T02:49:17.457 INFO:tasks.rados.rados.0.smithi191.stdout:272: rmattr oid 40 current snap is 21 2024-09-15T02:49:17.457 INFO:tasks.rados.rados.0.smithi191.stdout:272: done (0 left) 2024-09-15T02:49:17.458 INFO:tasks.rados.rados.0.smithi191.stdout:273: snap_create 2024-09-15T02:49:17.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:17.459 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:49:17.460 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:17.451+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:17.558 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:17.551+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:17.608 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:17.609 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:17.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:17.659 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:17.654+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:17.715 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:17.715 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:17.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:17.760 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:17.756+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:17.826 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.252627981000003, 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016202787999999999, 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568665+0000", 2024-09-15T02:49:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.568666+0000", 2024-09-15T02:49:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0699999999999997e-07 2024-09-15T02:49:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569158+0000", 2024-09-15T02:49:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000491633 2024-09-15T02:49:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569218+0000", 2024-09-15T02:49:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0742999999999998e-05 2024-09-15T02:49:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569231+0000", 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2455e-05 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569315+0000", 2024-09-15T02:49:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3701000000000004e-05 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569329+0000", 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4606e-05 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.584839+0000", 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015510222000000001 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.584868+0000", 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8821000000000001e-05 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.171679422, 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014988552, 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649614+0000", 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.649615+0000", 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8800000000000002e-07 2024-09-15T02:49:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650002+0000", 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038781599999999997 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650043+0000", 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0312999999999998e-05 2024-09-15T02:49:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650051+0000", 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2390000000000002e-06 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650118+0000", 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7015999999999997e-05 2024-09-15T02:49:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650127+0000", 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6200000000000005e-06 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664560+0000", 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014433692 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.664603+0000", 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2268000000000003e-05 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.12937067, 2024-09-15T02:49:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013640826, 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691923+0000", 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.691924+0000", 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.544e-06 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692182+0000", 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025800999999999998 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692284+0000", 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010141699999999999 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692295+0000", 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1447999999999999e-05 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692382+0000", 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7294999999999997e-05 2024-09-15T02:49:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.692401+0000", 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8635e-05 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.705531+0000", 2024-09-15T02:49:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013129403 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.705564+0000", 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3074000000000002e-05 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 49/33 ECSubWrite(tid=160, reqid=client.4526.0:364, at_version=49'23, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.124079395999999, 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011544153, 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697214+0000", 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697215+0000", 2024-09-15T02:49:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.046e-06 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697510+0000", 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00029446300000000002 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697550+0000", 2024-09-15T02:49:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0247999999999998e-05 2024-09-15T02:49:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.697562+0000", 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2068e-05 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.699512+0000", 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019506219999999999 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.699531+0000", 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8343999999999999e-05 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708741+0000", 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092103159999999996 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:53.708758+0000", 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7045999999999999e-05 2024-09-15T02:49:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.567720475000002, 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031206293, 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.253573+0000", 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254463+0000", 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088986800000000004 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254519+0000", 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5640000000000003e-05 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254529+0000", 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0152999999999999e-05 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254650+0000", 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012129300000000001 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.254679+0000", 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9107000000000001e-05 2024-09-15T02:49:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284735+0000", 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.030055693000000001 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284779+0000", 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4199000000000002e-05 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.560827239999998, 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024332761000000001, 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260466+0000", 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260468+0000", 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.508e-06 2024-09-15T02:49:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.260940+0000", 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047216899999999998 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.261103+0000", 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016333 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.261116+0000", 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.327e-05 2024-09-15T02:49:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.269681+0000", 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0085646730000000001 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.269691+0000", 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4339999999999998e-06 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284787+0000", 2024-09-15T02:49:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015096058000000001 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284799+0000", 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2319e-05 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.553245017999998, 2024-09-15T02:49:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01876924, 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268048+0000", 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268049+0000", 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268412+0000", 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000363111 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268461+0000", 2024-09-15T02:49:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9150000000000002e-05 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268470+0000", 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5909999999999996e-06 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268578+0000", 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107676 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.268591+0000", 2024-09-15T02:49:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3733e-05 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286797+0000", 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.018205193000000001 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.286818+0000", 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1012000000000001e-05 2024-09-15T02:49:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.1s1 50/33 ECSubWrite(tid=195, reqid=client.4526.0:411, at_version=50'34, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.551176441999999, 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014722531000000001, 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270117+0000", 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270118+0000", 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.012e-06 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270182+0000", 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4114000000000003e-05 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270229+0000", 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6473e-05 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.270241+0000", 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2092999999999999e-05 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275025+0000", 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047846529999999998 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275036+0000", 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0637e-05 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284820+0000", 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097838739999999997 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284840+0000", 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9675000000000001e-05 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.545881415, 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014093722, 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275412+0000", 2024-09-15T02:49:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275413+0000", 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1459999999999999e-06 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277178+0000", 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017643979999999999 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277259+0000", 2024-09-15T02:49:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1926000000000001e-05 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277268+0000", 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4249999999999998e-06 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277340+0000", 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2126999999999996e-05 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277360+0000", 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9678e-05 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289453+0000", 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012093051 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.289506+0000", 2024-09-15T02:49:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2970999999999998e-05 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as1 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.509167304999998, 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012325150999999999, 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312126+0000", 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312128+0000", 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.59e-06 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312511+0000", 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038326000000000001 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312619+0000", 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010794 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312637+0000", 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8104000000000001e-05 2024-09-15T02:49:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312798+0000", 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016069599999999999 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312831+0000", 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3179999999999997e-05 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324419+0000", 2024-09-15T02:49:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011588022999999999 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.324451+0000", 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2357999999999997e-05 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 60/33 ECSubWrite(tid=75, reqid=client.4526.0:529, at_version=60'52, trim_to=0'0, roll_forward_to=59'50))", 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.559385085000001, 2024-09-15T02:49:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014182907999999999, 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261908+0000", 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.261909+0000", 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262920+0000", 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001011236 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262962+0000", 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1980999999999997e-05 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.262988+0000", 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6571e-05 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263089+0000", 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100937 2024-09-15T02:49:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.263102+0000", 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3183999999999999e-05 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276063+0000", 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012960833 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.276091+0000", 2024-09-15T02:49:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8002999999999999e-05 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 61/53 ECSubWrite(tid=99, reqid=osd.14.0:99, at_version=61'43, trim_to=0'0, roll_forward_to=59'41))", 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.334519+0000", 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.486774222999999, 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014171448, 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334519+0000", 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334519+0000", 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334520+0000", 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2599999999999998e-07 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334563+0000", 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3136999999999997e-05 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334622+0000", 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9162e-05 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334629+0000", 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0380000000000002e-06 2024-09-15T02:49:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334682+0000", 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3012e-05 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334690+0000", 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5290000000000008e-06 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.348649+0000", 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013958834 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.348691+0000", 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1409999999999998e-05 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 61/33 ECSubWrite(tid=52, reqid=osd.15.0:52, at_version=61'40, trim_to=0'0, roll_forward_to=59'36))", 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:05.334691+0000", 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.486602129, 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016138692, 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334691+0000", 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334691+0000", 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334692+0000", 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3999999999999997e-07 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334707+0000", 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5034e-05 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334791+0000", 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4495999999999999e-05 2024-09-15T02:49:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.334796+0000", 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7829999999999998e-06 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.337673+0000", 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028774109999999999 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.337684+0000", 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0366e-05 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.350807+0000", 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013123374 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:05.350830+0000", 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2887999999999998e-05 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 62/33 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=62'48, trim_to=0'0, roll_forward_to=61'44))", 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:06.347594+0000", 2024-09-15T02:49:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.473699294999999, 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012425673999999999, 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347594+0000", 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347594+0000", 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347595+0000", 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5799999999999999e-07 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347620+0000", 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5384e-05 2024-09-15T02:49:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347751+0000", 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013113600000000001 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.347759+0000", 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6620000000000005e-06 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348433+0000", 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00067455 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.348451+0000", 2024-09-15T02:49:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7963999999999999e-05 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.359959+0000", 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011507969999999999 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:06.360020+0000", 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0449999999999999e-05 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 66/33 ECSubWrite(tid=64, reqid=client.4526.0:610, at_version=66'54, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.510782+0000", 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.3105116990000001, 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015282408000000001, 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510782+0000", 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510782+0000", 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.510783+0000", 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.567e-06 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512149+0000", 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001365502 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512202+0000", 2024-09-15T02:49:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3256e-05 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512213+0000", 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.075e-05 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512281+0000", 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7787999999999995e-05 2024-09-15T02:49:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.512299+0000", 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8754999999999999e-05 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.525981+0000", 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013681759999999999 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526064+0000", 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3029999999999996e-05 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 68/33 ECSubWrite(tid=78, reqid=client.4526.0:636, at_version=68'69, trim_to=0'0, roll_forward_to=68'67))", 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:12.499419+0000", 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.3218741270000001, 2024-09-15T02:49:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017615756, 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499419+0000", 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499419+0000", 2024-09-15T02:49:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499421+0000", 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8840000000000001e-06 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500186+0000", 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00076441499999999995 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500263+0000", 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7539000000000004e-05 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500276+0000", 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2916e-05 2024-09-15T02:49:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500369+0000", 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3270000000000007e-05 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500386+0000", 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6799999999999998e-05 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.516956+0000", 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016569971999999999 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.517035+0000", 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8960000000000003e-05 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 68/33 ECSubWrite(tid=81, reqid=client.4526.0:639, at_version=68'72, trim_to=0'0, roll_forward_to=68'67))", 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:12.522374+0000", 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.2989195130000004, 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15520476699999999, 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.522374+0000", 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.522374+0000", 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.522375+0000", 2024-09-15T02:49:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5199999999999996e-07 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.522395+0000", 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0288999999999998e-05 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.675784+0000", 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.153388887 2024-09-15T02:49:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.675796+0000", 2024-09-15T02:49:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2147000000000001e-05 2024-09-15T02:49:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.675936+0000", 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014037500000000001 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.675974+0000", 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7722999999999998e-05 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.677497+0000", 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015228069999999999 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.677579+0000", 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1786999999999995e-05 2024-09-15T02:49:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.bs5 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:16.274142+0000", 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.547151462, 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.021077762999999999, 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274142+0000", 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274142+0000", 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274143+0000", 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7899999999999998e-07 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274449+0000", 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030600899999999999 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274498+0000", 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9218999999999998e-05 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274506+0000", 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8839999999999994e-06 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274565+0000", 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9116000000000001e-05 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.274575+0000", 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8290000000000003e-06 2024-09-15T02:49:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.295191+0000", 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020616513 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.295220+0000", 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8513999999999998e-05 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s5 72/71 ECSubWrite(tid=98, reqid=osd.15.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=72'84))", 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:16.280400+0000", 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.540893182, 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014906780999999999, 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.280400+0000", 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.280400+0000", 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.280401+0000", 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.111e-06 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.280414+0000", 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2356e-05 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.281563+0000", 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011495399999999999 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.281573+0000", 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5100000000000004e-06 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.281629+0000", 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5835000000000003e-05 2024-09-15T02:49:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.281640+0000", 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1494999999999999e-05 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.295228+0000", 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013587494 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.295307+0000", 2024-09-15T02:49:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9439999999999998e-05 2024-09-15T02:49:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.bs5 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:16.283216+0000", 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.538077661, 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017532956999999998, 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283216+0000", 2024-09-15T02:49:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283216+0000", 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283217+0000", 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9299999999999996e-07 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283567+0000", 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035040700000000001 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283648+0000", 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0882000000000003e-05 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283657+0000", 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7260000000000008e-06 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283840+0000", 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018291299999999999 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.283850+0000", 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0875e-05 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.300712+0000", 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016861708 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.300749+0000", 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6553000000000001e-05 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:17.935 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:17.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:17.937 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:17.932+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:18.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:18.037 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:18.031+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:18.043 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:18.044 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:18.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:18.139 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:18.132+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:18.147 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:18.181 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:18.196 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-15T02:49:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-15T02:49:18.197 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:18.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:18.235 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:18.240 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:18.236+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:18.263 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:18.289 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:18.307 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-15T02:49:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-15T02:49:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-15T02:49:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:231 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.737514091999998, 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012963349000000001, 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 231 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565061+0000", 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565064+0000", 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1140000000000002e-06 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565078+0000", 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3900000000000001e-05 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565120+0000", 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2070999999999998e-05 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565135+0000", 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5085999999999999e-05 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565263+0000", 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012826600000000001 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565272+0000", 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0259999999999996e-06 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569405+0000", 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041323439999999996 2024-09-15T02:49:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.569560+0000", 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015537600000000001 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.570911+0000", 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013509349999999999 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574145+0000", 2024-09-15T02:49:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003233685 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.577972+0000", 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038275480000000001 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578024+0000", 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1997999999999999e-05 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:232 3.5s0 3:a906e5df:::smithi19125844-36:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.737410134000001, 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013333868, 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 232 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565165+0000", 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565166+0000", 2024-09-15T02:49:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.054e-06 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565170+0000", 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1969999999999998e-06 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565205+0000", 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4829999999999997e-05 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565234+0000", 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9320999999999999e-05 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565392+0000", 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015784999999999999 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565408+0000", 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5434000000000002e-05 2024-09-15T02:49:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571539+0000", 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061311990000000004 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.571658+0000", 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119486 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572348+0000", 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068956300000000002 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574147+0000", 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017987459999999999 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578458+0000", 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043109690000000004 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.578499+0000", 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1219e-05 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:233 3.5s0 3:a906e5df:::smithi19125844-36:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T02:49:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.737330141999998, 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.49017684700000003, 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 233 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565245+0000", 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565246+0000", 2024-09-15T02:49:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565250+0000", 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.76e-06 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565261+0000", 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1693e-05 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565268+0000", 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3570000000000003e-06 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565481+0000", 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021371699999999999 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.565537+0000", 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5838000000000003e-05 2024-09-15T02:49:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572781+0000", 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072440639999999997 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.572928+0000", 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014673400000000001 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.573788+0000", 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00086026199999999998 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582219+0000", 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084309350000000005 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055303+0000", 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.47308344099999999 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055422+0000", 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119119 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/33 ECSubWrite(tid=23, reqid=client.4526.0:236, at_version=39'18, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.728504536999999, 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089568029999999993, 2024-09-15T02:49:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574071+0000", 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6399999999999998e-07 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.574843+0000", 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00077236399999999999 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575043+0000", 2024-09-15T02:49:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019989399999999999 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575050+0000", 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.528e-06 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575124+0000", 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4041999999999993e-05 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.575134+0000", 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.582994+0000", 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0078592579999999992 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.583027+0000", 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3747000000000001e-05 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 39/33 ECSubWriteReply(tid=18, last_complete=39'11, committed=1, applied=1))", 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.726318335999999, 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.480047051, 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576257+0000", 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:43.576262+0000", 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.074e-06 2024-09-15T02:49:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055023+0000", 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.478760992 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055037+0000", 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3607e-05 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.055156+0000", 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118522 2024-09-15T02:49:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:44.056304+0000", 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001148154 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:249 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~704512] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.716934878000004, 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031968222999999997, 2024-09-15T02:49:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585640+0000", 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.585643+0000", 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7839999999999999e-06 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587905+0000", 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022617729999999999 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587945+0000", 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.981e-05 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.587959+0000", 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4297000000000001e-05 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588041+0000", 2024-09-15T02:49:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2514000000000003e-05 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588163+0000", 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012119 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589878+0000", 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017156809999999999 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607051+0000", 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017173134999999999 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617495+0000", 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010443661999999999 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617608+0000", 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113377 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:250 3.5s0 3:a210b161:::smithi19125844-29:head [append 0~376832] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.713656767000003, 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028844499999999999, 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 250 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588918+0000", 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.588920+0000", 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.807e-06 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589853+0000", 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00093252000000000005 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589887+0000", 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4721000000000002e-05 2024-09-15T02:49:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.589910+0000", 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2787000000000001e-05 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601237+0000", 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011326924 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.601404+0000", 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016697000000000001 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.602323+0000", 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00091864900000000001 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.607163+0000", 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048403409999999997 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617711+0000", 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010548119999999999 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:45.617763+0000", 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1660999999999999e-05 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:307 3.5s0 3:ae24c343:::smithi19125844-3:head [copy-from ver 10] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.683346460999999, 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.021447352999999999, 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 307 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619229+0000", 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619232+0000", 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619263+0000", 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1090000000000002e-05 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619289+0000", 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5284000000000001e-05 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619300+0000", 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1518000000000001e-05 2024-09-15T02:49:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619367+0000", 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6983000000000006e-05 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619450+0000", 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2785000000000001e-05 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.630087+0000", 2024-09-15T02:49:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010637047 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635664+0000", 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055774500000000003 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640607+0000", 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049425019999999997 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.640676+0000", 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9078000000000005e-05 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4526.0:308 3.5s0 3:ae24c343:::smithi19125844-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.683256825000001, 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016310021000000001, 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-15T02:49:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4526", 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 308 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619318+0000", 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619319+0000", 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9500000000000002e-07 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619324+0000", 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.87e-06 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619332+0000", 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2539999999999992e-06 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619335+0000", 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0079999999999998e-06 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619592+0000", 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000256724 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.619634+0000", 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2011999999999997e-05 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635446+0000", 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015811935999999999 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635533+0000", 2024-09-15T02:49:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7126000000000006e-05 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.635628+0000", 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5495999999999995e-05 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.652484851000001, 2024-09-15T02:49:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013723521000000001, 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650090+0000", 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650091+0000", 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04e-06 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650742+0000", 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000650436 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650792+0000", 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0680000000000003e-05 2024-09-15T02:49:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650812+0000", 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9397000000000001e-05 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650905+0000", 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3294000000000005e-05 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.650922+0000", 2024-09-15T02:49:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7209000000000001e-05 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663784+0000", 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012861490999999999 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:50.663814+0000", 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9974000000000001e-05 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 50/33 ECSubWrite(tid=37, reqid=client.4526.0:414, at_version=50'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.027157788, 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092230960000000001, 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.275417+0000", 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3300000000000001e-07 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276746+0000", 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013288589999999999 2024-09-15T02:49:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276893+0000", 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000146849 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276906+0000", 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2347000000000001e-05 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.276979+0000", 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3198e-05 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.277005+0000", 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5837e-05 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284582+0000", 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075769779999999998 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:54.284640+0000", 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8795000000000002e-05 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.375336592, 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009421937, 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927238+0000", 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927241+0000", 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0899999999999999e-06 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927476+0000", 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000235047 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927608+0000", 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013218999999999999 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927627+0000", 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.965e-05 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927797+0000", 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016982 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.927817+0000", 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9939999999999999e-05 2024-09-15T02:49:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936590+0000", 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087725600000000004 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:48:59.936660+0000", 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0640000000000001e-05 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.989835282000001, 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014263519000000001, 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312740+0000", 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.312741+0000", 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1999999999999998e-07 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313098+0000", 2024-09-15T02:49:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035754 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313251+0000", 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000153177 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313275+0000", 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3365000000000001e-05 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313599+0000", 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032383899999999997 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.313628+0000", 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.921e-05 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.326410+0000", 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012781957 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.327003+0000", 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00059361099999999997 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 57/33 ECSubWrite(tid=39, reqid=client.4526.0:482, at_version=57'24, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.973587041999998, 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010144892000000001, 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328988+0000", 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.328990+0000", 2024-09-15T02:49:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7099999999999999e-06 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329132+0000", 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000142068 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329222+0000", 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0607999999999997e-05 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329240+0000", 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7524e-05 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329313+0000", 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2658999999999997e-05 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.329323+0000", 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0497e-05 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339071+0000", 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097483840000000006 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:01.339133+0000", 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1441999999999999e-05 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 13.888749369999999, 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096704219999999997, 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413826+0000", 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7800000000000001e-07 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413927+0000", 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101127 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413988+0000", 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1103000000000001e-05 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.413997+0000", 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7709999999999995e-06 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414099+0000", 2024-09-15T02:49:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102274 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.414115+0000", 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6223e-05 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423433+0000", 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0093179969999999997 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:04.423496+0000", 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2749000000000005e-05 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 66/33 ECSubWrite(tid=66, reqid=client.4526.0:612, at_version=66'56, trim_to=0'0, roll_forward_to=66'52))", 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:10.526476+0000", 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 7.776098996, 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0094503, 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526476+0000", 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526476+0000", 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526477+0000", 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.082e-06 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526689+0000", 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021156699999999999 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526748+0000", 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9151000000000003e-05 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526757+0000", 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9209999999999998e-06 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526903+0000", 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014611 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.526915+0000", 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1953000000000001e-05 2024-09-15T02:49:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.535898+0000", 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089835350000000008 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:10.535926+0000", 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7980999999999999e-05 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.5s0 67/47 ECSubWrite(tid=349, reqid=osd.13.0:349, at_version=67'43, trim_to=0'0, roll_forward_to=66'39))", 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:11.419972+0000", 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.8826028729999997, 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.40161294400000003, 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419972+0000", 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419972+0000", 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419973+0000", 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3399999999999998e-07 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.419998+0000", 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4856000000000001e-05 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819425+0000", 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.39942740799999998 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819438+0000", 2024-09-15T02:49:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2401999999999999e-05 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819570+0000", 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000132423 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.819593+0000", 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2605000000000001e-05 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.821532+0000", 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00193954 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:11.821585+0000", 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2976000000000002e-05 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 68/33 ECSubWrite(tid=78, reqid=client.4526.0:636, at_version=68'69, trim_to=0'0, roll_forward_to=68'67))", 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:12.499554+0000", 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.8030208070000002, 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.018960830000000001, 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499554+0000", 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499554+0000", 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.499556+0000", 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5230000000000001e-06 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500443+0000", 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000886891 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500540+0000", 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7256000000000005e-05 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500559+0000", 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8626000000000001e-05 2024-09-15T02:49:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500719+0000", 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016086599999999999 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.500758+0000", 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8739999999999998e-05 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.518480+0000", 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017722011999999999 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.518515+0000", 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4916000000000002e-05 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 68/33 ECSubWrite(tid=79, reqid=client.4526.0:637, at_version=68'70, trim_to=0'0, roll_forward_to=68'67))", 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:12.507857+0000", 2024-09-15T02:49:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.794718069, 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014613673000000001, 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.507857+0000", 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.507857+0000", 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.507859+0000", 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.626e-06 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.508253+0000", 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00039394499999999998 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.508359+0000", 2024-09-15T02:49:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010688600000000001 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.508381+0000", 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1590000000000002e-05 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.512184+0000", 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038031610000000002 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.512204+0000", 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9334000000000002e-05 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.522446+0000", 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010242387 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:12.522471+0000", 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4743999999999999e-05 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.16s3 72/71 ECSubWrite(tid=418, reqid=osd.13.0:418, at_version=72'32, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-15T02:49:16.305429+0000", 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.997146112, 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.088281690999999995, 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.305429+0000", 2024-09-15T02:49:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.305429+0000", 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.305430+0000", 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8400000000000002e-07 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-15T02:49:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.305456+0000", 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.652e-05 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.392020+0000", 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.086563091999999994 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.392030+0000", 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0308e-05 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-15T02:49:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.392155+0000", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012538 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.392183+0000", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8242000000000002e-05 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.393625+0000", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014410530000000001 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-15T02:49:16.393711+0000", 2024-09-15T02:49:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6112000000000005e-05 2024-09-15T02:49:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-15T02:49:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-15T02:49:18.407 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:18.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:18.409 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:18.409 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:18.410 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:18.410 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:18.410 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:18.403+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:18.429 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:18.440 INFO:tasks.rados.rados.0.smithi191.stdout:273: done (0 left) 2024-09-15T02:49:18.440 INFO:tasks.rados.rados.0.smithi191.stdout:274: read oid 11 snap 5 2024-09-15T02:49:18.440 INFO:tasks.rados.rados.0.smithi191.stdout:274: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:49:18.440 INFO:tasks.rados.rados.0.smithi191.stdout:275: read oid 29 snap -1 2024-09-15T02:49:18.440 INFO:tasks.rados.rados.0.smithi191.stdout:275: expect deleted 2024-09-15T02:49:18.440 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 26 current snap is 22 2024-09-15T02:49:18.441 INFO:tasks.rados.rados.0.smithi191.stdout:276: seq_num 111 ranges {0=704512,704512=655360,1359872=507904} 2024-09-15T02:49:18.458 INFO:tasks.rados.rados.0.smithi191.stdout:276: writing smithi19125844-26 from 0 to 704512 tid 1 2024-09-15T02:49:18.463 INFO:tasks.rados.rados.0.smithi191.stdout:276: writing smithi19125844-26 from 704512 to 1359872 tid 2 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:276: writing smithi19125844-26 from 1359872 to 1867776 tid 3 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:277: rollback oid 16 current snap is 22 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 16 to 19 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:275: done (3 left) 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:278: rollback oid 38 current snap is 22 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 38 to 18 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:279: read oid 3 snap -1 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:279: expect deleted 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:280: copy_from oid 18 from oid 29 current snap is 22 2024-09-15T02:49:18.469 INFO:tasks.rados.rados.0.smithi191.stdout:281: read oid 20 snap -1 2024-09-15T02:49:18.470 INFO:tasks.rados.rados.0.smithi191.stdout:281: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:49:18.470 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 6 current snap is 22 2024-09-15T02:49:18.470 INFO:tasks.rados.rados.0.smithi191.stdout:282: seq_num 112 ranges {0=720896,720896=802816,1523712=360448} 2024-09-15T02:49:18.470 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:18.474 INFO:tasks.rados.rados.0.smithi191.stdout:282: writing smithi19125844-6 from 0 to 720896 tid 1 2024-09-15T02:49:18.481 INFO:tasks.rados.rados.0.smithi191.stdout:282: writing smithi19125844-6 from 720896 to 1523712 tid 2 2024-09-15T02:49:18.483 INFO:tasks.rados.rados.0.smithi191.stdout:282: writing smithi19125844-6 from 1523712 to 1884160 tid 3 2024-09-15T02:49:18.484 INFO:tasks.rados.rados.0.smithi191.stdout:283: copy_from oid 8 from oid 39 current snap is 22 2024-09-15T02:49:18.484 INFO:tasks.rados.rados.0.smithi191.stdout:281: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:49:18.501 INFO:tasks.rados.rados.0.smithi191.stdout:274: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T02:49:18.509 INFO:tasks.rados.rados.0.smithi191.stdout:280: finishing copy_from to smithi19125844-18 2024-09-15T02:49:18.509 INFO:tasks.rados.rados.0.smithi191.stdout:280: got expected ENOENT (src dne) 2024-09-15T02:49:18.509 INFO:tasks.rados.rados.0.smithi191.stdout:280: finishing copy_from racing read to smithi19125844-18 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:276: finishing write tid 1 to smithi19125844-26 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:276: finishing write tid 2 to smithi19125844-26 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:276: finishing write tid 3 to smithi19125844-26 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:276: finishing write tid 4 to smithi19125844-26 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:274: done (8 left) 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:280: done (7 left) 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:281: done (6 left) 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:284: copy_from oid 43 from oid 28 current snap is 22 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:285: read oid 14 snap -1 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:285: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:286: rollback oid 42 current snap is 22 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 42 to 17 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:287: read oid 15 snap -1 2024-09-15T02:49:18.510 INFO:tasks.rados.rados.0.smithi191.stdout:287: expect deleted 2024-09-15T02:49:18.511 INFO:tasks.rados.rados.0.smithi191.stdout:288: rmattr oid 19 current snap is 22 2024-09-15T02:49:18.511 INFO:tasks.rados.rados.0.smithi191.stdout:288: done (10 left) 2024-09-15T02:49:18.511 INFO:tasks.rados.rados.0.smithi191.stdout:289: read oid 20 snap 19 2024-09-15T02:49:18.511 INFO:tasks.rados.rados.0.smithi191.stdout:289: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:49:18.511 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:18.503+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:18.512 INFO:tasks.rados.rados.0.smithi191.stdout:290: read oid 30 snap -1 2024-09-15T02:49:18.512 INFO:tasks.rados.rados.0.smithi191.stdout:290: expect (ObjNum 102 snap 18 seq_num 102) 2024-09-15T02:49:18.512 INFO:tasks.rados.rados.0.smithi191.stdout:291: copy_from oid 47 from oid 48 current snap is 22 2024-09-15T02:49:18.512 INFO:tasks.rados.rados.0.smithi191.stdout:292: snap_create 2024-09-15T02:49:18.512 INFO:tasks.rados.rados.0.smithi191.stdout:276: finishing write tid 5 to smithi19125844-26 2024-09-15T02:49:18.513 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 26 v 59 (ObjNum 111 snap 22 seq_num 111) dirty exists 2024-09-15T02:49:18.513 INFO:tasks.rados.rados.0.smithi191.stdout:276: left oid 26 (ObjNum 111 snap 22 seq_num 111) 2024-09-15T02:49:18.513 INFO:tasks.rados.rados.0.smithi191.stdout:282: finishing write tid 1 to smithi19125844-6 2024-09-15T02:49:18.519 INFO:tasks.rados.rados.0.smithi191.stdout:289: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T02:49:18.519 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:278: finishing rollback tid 0 to smithi19125844-38 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:286: finishing rollback tid 0 to smithi19125844-42 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:286: finishing rollback tid 1 to smithi19125844-42 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:286: finishing rollback tid 2 to smithi19125844-42 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 42 v 23 (ObjNum 96 snap 15 seq_num 96) dirty exists 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:278: finishing rollback tid 1 to smithi19125844-38 2024-09-15T02:49:18.541 INFO:tasks.rados.rados.0.smithi191.stdout:290: expect (ObjNum 102 snap 18 seq_num 102) 2024-09-15T02:49:18.543 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:18.579 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:18.579 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:18.581 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:18.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:18.610 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:18.604+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:18.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:18.603+0000 7f9bb7fff700 0 -- 172.21.15.6:0/2516081465 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7f9bc0195440 msgr2=0x7f9bc0197820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:282: finishing write tid 2 to smithi19125844-6 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:282: finishing write tid 3 to smithi19125844-6 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:277: finishing rollback tid 1 to smithi19125844-16 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 16 v 0 (ObjNum 161 snap 0 seq_num 1) dirty dne 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:278: finishing rollback tid 2 to smithi19125844-38 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 38 v 46 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:291: finishing copy_from to smithi19125844-47 2024-09-15T02:49:18.617 INFO:tasks.rados.rados.0.smithi191.stdout:291: got expected ENOENT (src dne) 2024-09-15T02:49:18.618 INFO:tasks.rados.rados.0.smithi191.stdout:291: finishing copy_from racing read to smithi19125844-47 2024-09-15T02:49:18.618 INFO:tasks.rados.rados.0.smithi191.stdout:285: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T02:49:18.618 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:284: finishing copy_from racing read to smithi19125844-43 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:282: finishing write tid 4 to smithi19125844-6 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:282: finishing write tid 5 to smithi19125844-6 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 6 v 63 (ObjNum 112 snap 22 seq_num 112) dirty exists 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:282: left oid 6 (ObjNum 112 snap 22 seq_num 112) 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:284: finishing copy_from to smithi19125844-43 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 44 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:283: finishing copy_from racing read to smithi19125844-8 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:283: finishing copy_from to smithi19125844-8 2024-09-15T02:49:18.620 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 8 v 25 (ObjNum 77 snap 8 seq_num 77) dirty exists 2024-09-15T02:49:18.648 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:18.650 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:18.706 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:18.707 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:18.711 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:18.704+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:18.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:18.812 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:18.808+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:18.817 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:18.817 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:18.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:18.913 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:18.908+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:18.937 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:18.937 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:18.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:19.014 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:19.008+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:19.052 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.052 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:19.115 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:19.108+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:19.154 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.155 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:19.216 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:19.208+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:19.260 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.260 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.261 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 10] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:49:19.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:49:19.262 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-15T02:49:19.262 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-15T02:49:19.262 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-15T02:49:19.262 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:49:19.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:19.317 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:19.312+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:19.335 INFO:tasks.ceph.osd.6:Stopped 2024-09-15T02:49:19.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 6 2024-09-15T02:49:19.367 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.367 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:19.418 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:19.412+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:19.442 INFO:tasks.rados.rados.0.smithi191.stdout:276: done (13 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:277: done (12 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:278: done (11 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:279: done (10 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:282: done (9 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:283: done (8 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:284: done (7 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:285: done (6 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:286: done (5 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:287: done (4 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:289: done (3 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:290: done (2 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:291: done (1 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:292: done (0 left) 2024-09-15T02:49:19.443 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 31 current snap is 23 2024-09-15T02:49:19.444 INFO:tasks.rados.rados.0.smithi191.stdout:293: seq_num 113 ranges {507904=737280,1245184=557056,1802240=606208} 2024-09-15T02:49:19.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:49:19.464 INFO:tasks.rados.rados.0.smithi191.stdout:293: writing smithi19125844-31 from 507904 to 1245184 tid 1 2024-09-15T02:49:19.469 INFO:tasks.rados.rados.0.smithi191.stdout:293: writing smithi19125844-31 from 1245184 to 1802240 tid 2 2024-09-15T02:49:19.474 INFO:tasks.rados.rados.0.smithi191.stdout:293: writing smithi19125844-31 from 1802240 to 2408448 tid 3 2024-09-15T02:49:19.474 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 48 current snap is 23 2024-09-15T02:49:19.474 INFO:tasks.rados.rados.0.smithi191.stdout:294: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2024-09-15T02:49:19.478 INFO:tasks.rados.rados.0.smithi191.stdout:294: writing smithi19125844-48 from 0 to 753664 tid 1 2024-09-15T02:49:19.479 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.479 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.482 INFO:tasks.rados.rados.0.smithi191.stdout:294: writing smithi19125844-48 from 753664 to 1441792 tid 2 2024-09-15T02:49:19.485 INFO:tasks.rados.rados.0.smithi191.stdout:294: writing smithi19125844-48 from 1441792 to 1916928 tid 3 2024-09-15T02:49:19.485 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 1 current snap is 23 2024-09-15T02:49:19.486 INFO:tasks.rados.rados.0.smithi191.stdout:295: seq_num 115 ranges {3817472=770048,4587520=704512,5292032=458752} 2024-09-15T02:49:19.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:19.512 INFO:tasks.rados.rados.0.smithi191.stdout:295: writing smithi19125844-1 from 3817472 to 4587520 tid 1 2024-09-15T02:49:19.517 INFO:tasks.rados.rados.0.smithi191.stdout:295: writing smithi19125844-1 from 4587520 to 5292032 tid 2 2024-09-15T02:49:19.519 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:19.511+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:19.522 INFO:tasks.rados.rados.0.smithi191.stdout:295: writing smithi19125844-1 from 5292032 to 5750784 tid 3 2024-09-15T02:49:19.522 INFO:tasks.rados.rados.0.smithi191.stdout:296: rmattr oid 19 current snap is 23 2024-09-15T02:49:19.522 INFO:tasks.rados.rados.0.smithi191.stdout:296: done (3 left) 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:293: finishing write tid 1 to smithi19125844-31 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:293: finishing write tid 2 to smithi19125844-31 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:293: finishing write tid 3 to smithi19125844-31 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:297: read oid 4 snap -1 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:297: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:298: copy_from oid 17 from oid 8 current snap is 23 2024-09-15T02:49:19.523 INFO:tasks.rados.rados.0.smithi191.stdout:299: rollback oid 47 current snap is 23 2024-09-15T02:49:19.524 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 47 to 5 2024-09-15T02:49:19.524 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 44 current snap is 23 2024-09-15T02:49:19.524 INFO:tasks.rados.rados.0.smithi191.stdout:300: seq_num 116 ranges {2392064=786432,3178496=458752,3637248=442368,4079616=262144} 2024-09-15T02:49:19.541 INFO:tasks.rados.rados.0.smithi191.stdout:300: writing smithi19125844-44 from 2392064 to 3178496 tid 1 2024-09-15T02:49:19.544 INFO:tasks.rados.rados.0.smithi191.stdout:300: writing smithi19125844-44 from 3178496 to 3637248 tid 2 2024-09-15T02:49:19.547 INFO:tasks.rados.rados.0.smithi191.stdout:300: writing smithi19125844-44 from 3637248 to 4079616 tid 3 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:300: writing smithi19125844-44 from 4079616 to 4341760 tid 4 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:301: read oid 41 snap -1 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:301: expect (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:302: rollback oid 38 current snap is 23 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 38 to 20 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 30 current snap is 23 2024-09-15T02:49:19.550 INFO:tasks.rados.rados.0.smithi191.stdout:303: seq_num 117 ranges {2211840=425984,2637824=606208,3244032=540672,3784704=409600} 2024-09-15T02:49:19.564 INFO:tasks.rados.rados.0.smithi191.stdout:303: writing smithi19125844-30 from 2211840 to 2637824 tid 1 2024-09-15T02:49:19.568 INFO:tasks.rados.rados.0.smithi191.stdout:303: writing smithi19125844-30 from 2637824 to 3244032 tid 2 2024-09-15T02:49:19.572 INFO:tasks.rados.rados.0.smithi191.stdout:303: writing smithi19125844-30 from 3244032 to 3784704 tid 3 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:303: writing smithi19125844-30 from 3784704 to 4194304 tid 4 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:304: snap_create 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:295: finishing write tid 1 to smithi19125844-1 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:295: finishing write tid 2 to smithi19125844-1 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:295: finishing write tid 3 to smithi19125844-1 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:295: finishing write tid 4 to smithi19125844-1 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:295: finishing write tid 5 to smithi19125844-1 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 1 v 72 (ObjNum 115 snap 23 seq_num 115) dirty exists 2024-09-15T02:49:19.576 INFO:tasks.rados.rados.0.smithi191.stdout:295: left oid 1 (ObjNum 115 snap 23 seq_num 115) 2024-09-15T02:49:19.593 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.593 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.594 INFO:tasks.rados.rados.0.smithi191.stdout:303: finishing write tid 1 to smithi19125844-30 2024-09-15T02:49:19.595 INFO:tasks.rados.rados.0.smithi191.stdout:303: finishing write tid 2 to smithi19125844-30 2024-09-15T02:49:19.595 INFO:tasks.rados.rados.0.smithi191.stdout:303: finishing write tid 3 to smithi19125844-30 2024-09-15T02:49:19.595 INFO:tasks.rados.rados.0.smithi191.stdout:303: finishing write tid 4 to smithi19125844-30 2024-09-15T02:49:19.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:19.620 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:19.612+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:19.693 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.694 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:19.706 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-15T02:49:19.699+0000 7f08e39de700 -1 mon.a@0(leader).osd e75 definitely_dead 0 2024-09-15T02:49:19.710 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6 is already down. 2024-09-15T02:49:19.721 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 10] 2024-09-15T02:49:19.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-09-15T02:49:19.748 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:19.744+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:19.803 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.804 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:19.823 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:19.818+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:19.913 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:19.914 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:19.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:19.924 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:19.915+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:20.019 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.020 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.022 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-15T02:49:20.015+0000 7f08e39de700 -1 mon.a@0(leader).osd e75 definitely_dead 0 2024-09-15T02:49:20.023 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:20.015+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:20.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:20.124 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:20.119+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:20.125 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.125 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.134 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:20.225 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:20.223+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:20.229 INFO:tasks.rados.rados.0.smithi191.stdout:303: finishing write tid 5 to smithi19125844-30 2024-09-15T02:49:20.233 INFO:tasks.rados.rados.0.smithi191.stdout:303: finishing write tid 6 to smithi19125844-30 2024-09-15T02:49:20.233 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 30 v 90 (ObjNum 117 snap 23 seq_num 117) dirty exists 2024-09-15T02:49:20.233 INFO:tasks.rados.rados.0.smithi191.stdout:303: left oid 30 (ObjNum 117 snap 23 seq_num 117) 2024-09-15T02:49:20.243 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.244 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.253 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:20.327 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:20.322+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:20.356 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.357 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.366 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:20.428 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:20.419+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:20.443 INFO:tasks.rados.rados.0.smithi191.stdout:295: done (10 left) 2024-09-15T02:49:20.443 INFO:tasks.rados.rados.0.smithi191.stdout:303: done (9 left) 2024-09-15T02:49:20.443 INFO:tasks.rados.rados.0.smithi191.stdout:304: done (8 left) 2024-09-15T02:49:20.444 INFO:tasks.rados.rados.0.smithi191.stdout:305: read oid 10 snap 17 2024-09-15T02:49:20.444 INFO:tasks.rados.rados.0.smithi191.stdout:305: expect (ObjNum 93 snap 15 seq_num 93) 2024-09-15T02:49:20.444 INFO:tasks.rados.rados.0.smithi191.stdout:306: snap_create 2024-09-15T02:49:20.447 INFO:tasks.rados.rados.0.smithi191.stdout:293: finishing write tid 4 to smithi19125844-31 2024-09-15T02:49:20.456 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:49:20.456 INFO:tasks.rados.rados.0.smithi191.stdout:293: finishing write tid 5 to smithi19125844-31 2024-09-15T02:49:20.456 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 31 v 68 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-09-15T02:49:20.457 INFO:tasks.rados.rados.0.smithi191.stdout:293: left oid 31 (ObjNum 113 snap 23 seq_num 113) 2024-09-15T02:49:20.460 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.6. 2024-09-15T02:49:20.472 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-09-15T02:49:20.472 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:49:20.473 DEBUG:teuthology.orchestra.run.smithi152:> type ceph-objectstore-tool 2024-09-15T02:49:20.474 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.474 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.480 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T02:49:20.481 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-09-15T02:49:20.484 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:20.528 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:20.520+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:20.629 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:20.624+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:20.630 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.630 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.639 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:20.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:20.730 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:20.724+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:20.735 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.735 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.744 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:20.796 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:49:20.794+0000 7f3703cc0300 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:49:20.822 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:20.831 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:20.824+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:20.840 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:20.840 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:20.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:20.851 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:20.885 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:20.914 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:20.932 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:20.924+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:20.942 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:20.959 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:20.959 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:20.959 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:20.959 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:20.968 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:20.971 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:21.000 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:21.029 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:21.032 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:21.024+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:21.058 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:21.071 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:21.072 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:21.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:21.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:21.113 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:21.133 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:21.127+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:21.142 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:21.171 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:21.202 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:21.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:21.203+0000 7f196effd700 0 -- 172.21.15.6:0/3418308611 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7f19781935c0 msgr2=0x7f19781959e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:21.220 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:21.220 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:21.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 37.647307259999998, 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:21.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 37.637364953999999, 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099046280000000004, 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579386+0000", 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.536e-06 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579616+0000", 2024-09-15T02:49:21.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230366 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579660+0000", 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4489e-05 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579668+0000", 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4800000000000004e-06 2024-09-15T02:49:21.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579731+0000", 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2733999999999999e-05 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579742+0000", 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589278+0000", 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095358219999999994 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589289+0000", 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0596e-05 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.230 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.566360578000001, 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015141226000000001, 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650389+0000", 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650600+0000", 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021155999999999999 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650679+0000", 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8641999999999997e-05 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650693+0000", 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4294000000000001e-05 2024-09-15T02:49:21.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650801+0000", 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108132 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650828+0000", 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6565999999999999e-05 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665503+0000", 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014675142 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665530+0000", 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6509999999999999e-05 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.235 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=164, reqid=client.4526.0:313, at_version=46'24, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.658092+0000", 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.558656641999999, 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081368880000000001, 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.658092+0000", 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.658092+0000", 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.658093+0000", 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1179999999999999e-06 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.658210+0000", 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116888 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.658264+0000", 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4083999999999999e-05 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.658273+0000", 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8570000000000001e-06 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.662130+0000", 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003857199 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.662139+0000", 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0949999999999996e-06 2024-09-15T02:49:21.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666199+0000", 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004060513 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666229+0000", 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0134e-05 2024-09-15T02:49:21.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 49/33 ECSubWrite(tid=175, reqid=client.4526.0:354, at_version=49'27, trim_to=0'0, roll_forward_to=46'25))", 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.668163+0000", 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.548585639999999, 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067601079999999999, 2024-09-15T02:49:21.241 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668163+0000", 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668163+0000", 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668164+0000", 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1999999999999998e-07 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668386+0000", 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022177399999999999 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668530+0000", 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014395900000000001 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668551+0000", 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1231999999999999e-05 2024-09-15T02:49:21.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668736+0000", 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184728 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.668773+0000", 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6667e-05 2024-09-15T02:49:21.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.674876+0000", 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061034849999999996 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.674923+0000", 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7443e-05 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.246 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.524266344000001, 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011398508, 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.22e-07 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693204+0000", 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072156 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693258+0000", 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3134999999999999e-05 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693266+0000", 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5259999999999993e-06 2024-09-15T02:49:21.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693333+0000", 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6580000000000003e-05 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693344+0000", 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0971999999999999e-05 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703854+0000", 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010510265 2024-09-15T02:49:21.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703881+0000", 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7248000000000001e-05 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.694524+0000", 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.522224855000001, 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091159489999999999, 2024-09-15T02:49:21.251 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694524+0000", 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694524+0000", 2024-09-15T02:49:21.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694525+0000", 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8699999999999995e-07 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695089+0000", 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00056474900000000002 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695146+0000", 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6920999999999999e-05 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695157+0000", 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0828e-05 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695241+0000", 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3974999999999995e-05 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695254+0000", 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2405999999999999e-05 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703597+0000", 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083438360000000003 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703640+0000", 2024-09-15T02:49:21.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2647e-05 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=160, reqid=client.4526.0:364, at_version=49'23, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.698222+0000", 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.518526509000001, 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0086224960000000003, 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.257 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698222+0000", 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698222+0000", 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698223+0000", 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2900000000000002e-07 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698661+0000", 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00043797500000000001 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698706+0000", 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4914999999999999e-05 2024-09-15T02:49:21.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698714+0000", 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.699072+0000", 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035784400000000001 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.699084+0000", 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2413e-05 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706820+0000", 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077354219999999996 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706845+0000", 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5066e-05 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=161, reqid=client.4526.0:365, at_version=49'24, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.702878+0000", 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.513871356999999, 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0090111389999999996, 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.262 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702878+0000", 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702878+0000", 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702879+0000", 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1599999999999999e-06 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702998+0000", 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118973 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703046+0000", 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8607999999999999e-05 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703058+0000", 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1574e-05 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703916+0000", 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00085808100000000001 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703927+0000", 2024-09-15T02:49:21.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0798000000000001e-05 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.711856+0000", 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079289029999999993 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.711889+0000", 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3042e-05 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.266 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=164, reqid=client.4526.0:366, at_version=49'25, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.705193+0000", 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.511555740999999, 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067151650000000004, 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705193+0000", 2024-09-15T02:49:21.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705193+0000", 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705194+0000", 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3900000000000003e-07 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705216+0000", 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.154e-05 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705264+0000", 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8161999999999997e-05 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705327+0000", 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3372000000000002e-05 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706642+0000", 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013146500000000001 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706654+0000", 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1698e-05 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.711896+0000", 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052425609999999997 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.711908+0000", 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2243e-05 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=171, reqid=client.4526.0:380, at_version=49'28, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:21.272 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.725864+0000", 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.490885174999999, 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075756069999999998, 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.725864+0000", 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.725864+0000", 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.725865+0000", 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3900000000000003e-07 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.726167+0000", 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030235499999999997 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.726205+0000", 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7472999999999998e-05 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.726213+0000", 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6969999999999999e-06 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.726276+0000", 2024-09-15T02:49:21.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2594000000000004e-05 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.726287+0000", 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1548e-05 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.733364+0000", 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070762009999999998 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.733439+0000", 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5799999999999999e-05 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 26.962435715000002, 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016572159999999999, 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.277 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254314+0000", 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1100000000000002e-07 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255086+0000", 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077269100000000005 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255147+0000", 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1005999999999998e-05 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255166+0000", 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8581999999999999e-05 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255267+0000", 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100973 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255289+0000", 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2565000000000002e-05 2024-09-15T02:49:21.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270855+0000", 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015565733999999999 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270885+0000", 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0298e-05 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.261882+0000", 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 26.954866579000001, 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0094284799999999995, 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261882+0000", 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261882+0000", 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261883+0000", 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9600000000000008e-07 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261996+0000", 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113041 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262066+0000", 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9851999999999995e-05 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262078+0000", 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1591e-05 2024-09-15T02:49:21.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.265641+0000", 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003562843 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.265656+0000", 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5531000000000001e-05 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.271284+0000", 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056282629999999997 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.271311+0000", 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6363000000000002e-05 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 26.947944759999999, 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018797873, 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8999999999999998e-07 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269160+0000", 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035577800000000001 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3349000000000001e-05 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269211+0000", 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9439999999999999e-06 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269283+0000", 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2263999999999998e-05 2024-09-15T02:49:21.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269296+0000", 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3145e-05 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287519+0000", 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018222955999999998 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287602+0000", 2024-09-15T02:49:21.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3146999999999995e-05 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 26.939672133999998, 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010563233999999999, 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277078+0000", 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1500000000000004e-07 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278860+0000", 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017825339999999999 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278958+0000", 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7705000000000001e-05 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278971+0000", 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3481e-05 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279037+0000", 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5848999999999999e-05 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279045+0000", 2024-09-15T02:49:21.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6359999999999994e-06 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287611+0000", 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085663470000000002 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287640+0000", 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8966999999999999e-05 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.295 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 51/33 ECSubWrite(tid=66, reqid=client.4526.0:443, at_version=51'9, trim_to=0'0, roll_forward_to=50'7))", 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:55.235922+0000", 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.980827024, 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075054609999999997, 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.235922+0000", 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.235922+0000", 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.235923+0000", 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8999999999999996e-07 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.235987+0000", 2024-09-15T02:49:21.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4382999999999997e-05 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.236110+0000", 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000122688 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.236129+0000", 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.931e-05 2024-09-15T02:49:21.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.236283+0000", 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015375499999999999 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.236311+0000", 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8180000000000001e-05 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.243366+0000", 2024-09-15T02:49:21.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070550559999999997 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:55.243427+0000", 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1398999999999999e-05 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.300 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.288937690000001, 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010079322999999999, 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927813+0000", 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9630000000000001e-06 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928385+0000", 2024-09-15T02:49:21.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057185600000000004 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928464+0000", 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8615999999999994e-05 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928480+0000", 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6696000000000001e-05 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928602+0000", 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121209 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928622+0000", 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.052e-05 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937813+0000", 2024-09-15T02:49:21.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091904280000000005 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937891+0000", 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8034999999999994e-05 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 57/33 ECSubWrite(tid=38, reqid=client.4526.0:481, at_version=57'23, trim_to=0'0, roll_forward_to=55'21))", 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:01.312770+0000", 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.903978597999998, 2024-09-15T02:49:21.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0088263739999999997, 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.312770+0000", 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.312770+0000", 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.312771+0000", 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5300000000000003e-07 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.313287+0000", 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051626700000000005 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.313347+0000", 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9129999999999998e-05 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.313361+0000", 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4413000000000001e-05 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.313488+0000", 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126754 2024-09-15T02:49:21.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.313514+0000", 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5743000000000001e-05 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.321557+0000", 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0080434949999999995 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:01.321597+0000", 2024-09-15T02:49:21.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9718999999999997e-05 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 72/71 ECSubWrite(tid=403, reqid=osd.13.0:403, at_version=72'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.258998+0000", 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.9577513709999996, 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0086092630000000007, 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.258998+0000", 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.258998+0000", 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.258998+0000", 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4400000000000001e-07 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.259020+0000", 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1832e-05 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.259093+0000", 2024-09-15T02:49:21.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2916000000000005e-05 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.259098+0000", 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3329999999999996e-06 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.259496+0000", 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039763199999999998 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.259504+0000", 2024-09-15T02:49:21.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8399999999999995e-06 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.267580+0000", 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0080766210000000008 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.267607+0000", 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6645e-05 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(3.1s2 RELEASE e76)", 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.54711422099999996, 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43008043499999998, 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669637+0000", 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.917e-06 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669648+0000", 2024-09-15T02:49:21.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1653e-05 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099516+0000", 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42986811200000002 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099534+0000", 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7997000000000001e-05 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099653+0000", 2024-09-15T02:49:21.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011822100000000001 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099681+0000", 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8496999999999998e-05 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099715+0000", 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4038000000000002e-05 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.318 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.319 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.319 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.319 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:21.320 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:21.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:21.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:21.324 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:21.316+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:21.421 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:21.418+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:21.429 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:21.430 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:21.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:21.457 INFO:tasks.rados.rados.0.smithi191.stdout:293: done (9 left) 2024-09-15T02:49:21.457 INFO:tasks.rados.rados.0.smithi191.stdout:306: done (8 left) 2024-09-15T02:49:21.457 INFO:tasks.rados.rados.0.smithi191.stdout:307: snap_remove snap 24 2024-09-15T02:49:21.521 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:21.516+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:21.579 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:21.579 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:21.579 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:21.579 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:21.579 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:21.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:21.633 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:21.628+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:21.723 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:21.715+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:21.739 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:21.739 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:21.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:21.739 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.165901150000003, 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.153384965000001, 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:49:21.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:49:21.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:49:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:21.749 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.152345805000003, 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:49:21.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:49:21.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:49:21.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.755 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 32.101723436999997, 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010839624000000001, 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.633919+0000", 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.17e-07 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634471+0000", 2024-09-15T02:49:21.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00055238900000000003 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634518+0000", 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6437000000000003e-05 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634530+0000", 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1926999999999999e-05 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634612+0000", 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2633000000000007e-05 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.634626+0000", 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4088e-05 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.644722+0000", 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010095615 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:49.644758+0000", 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418000000000003e-05 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 31.084594799000001, 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015031548, 2024-09-15T02:49:21.760 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651048+0000", 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4900000000000001e-07 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651266+0000", 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000218233 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651324+0000", 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8544000000000001e-05 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651335+0000", 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336000000000001e-05 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651408+0000", 2024-09-15T02:49:21.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3415000000000001e-05 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651424+0000", 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6001000000000001e-05 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666045+0000", 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014620861000000001 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666079+0000", 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3809000000000001e-05 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:21.765 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.043222414999999, 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015570855999999999, 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:21.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692736+0000", 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031561799999999999 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692809+0000", 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3174000000000001e-05 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692821+0000", 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1627000000000001e-05 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692906+0000", 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5172000000000004e-05 2024-09-15T02:49:21.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692920+0000", 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4361e-05 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707954+0000", 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015033451999999999 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707991+0000", 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6714000000000001e-05 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as2 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.040921211000001, 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011075066, 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.770 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9400000000000001e-07 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695264+0000", 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054240199999999999 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695308+0000", 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4595999999999998e-05 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695316+0000", 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.605e-06 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695420+0000", 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000104505 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695441+0000", 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.084e-05 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705756+0000", 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010315094 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705796+0000", 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9629999999999998e-05 2024-09-15T02:49:21.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=160, reqid=client.4526.0:364, at_version=49'23, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.698119+0000", 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.037522783, 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099049519999999999, 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698119+0000", 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698119+0000", 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698120+0000", 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.61e-07 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698817+0000", 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069735800000000003 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698866+0000", 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8718999999999999e-05 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698874+0000", 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2390000000000002e-06 2024-09-15T02:49:21.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698948+0000", 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3779e-05 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.698964+0000", 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5951e-05 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.708004+0000", 2024-09-15T02:49:21.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0090398860000000004 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.708024+0000", 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0259e-05 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.986160259999998, 2024-09-15T02:49:21.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014530774999999999, 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749483+0000", 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4799999999999997e-07 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749717+0000", 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000234408 2024-09-15T02:49:21.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749768+0000", 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1155000000000001e-05 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749777+0000", 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1609999999999992e-06 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749853+0000", 2024-09-15T02:49:21.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5803000000000006e-05 2024-09-15T02:49:21.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749872+0000", 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556e-05 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.763971+0000", 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014098934 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.764013+0000", 2024-09-15T02:49:21.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1810000000000001e-05 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.481279833999999, 2024-09-15T02:49:21.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012056918, 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:21.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254363+0000", 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7199999999999997e-07 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254896+0000", 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053267899999999999 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255034+0000", 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013821100000000001 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255066+0000", 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255135+0000", 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8885000000000002e-05 2024-09-15T02:49:21.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255148+0000", 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2498e-05 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266383+0000", 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011235187000000001 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266419+0000", 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6358e-05 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.444505229000001, 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011905962000000001, 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291138+0000", 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291342+0000", 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203403 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291572+0000", 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230084 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291579+0000", 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6990000000000003e-06 2024-09-15T02:49:21.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291641+0000", 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2093000000000003e-05 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291650+0000", 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303018+0000", 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011368296 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303043+0000", 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4901000000000001e-05 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.794 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.444270839000001, 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011692335, 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291548+0000", 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000177021 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291673+0000", 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012469299999999999 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291678+0000", 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296939+0000", 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052609120000000004 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296948+0000", 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303051+0000", 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006102433 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303064+0000", 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2948999999999999e-05 2024-09-15T02:49:21.798 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/33 ECSubWrite(tid=212, reqid=client.4526.0:450, at_version=52'22, trim_to=0'0, roll_forward_to=49'21))", 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.487182656000002, 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015878961, 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248460+0000", 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249338+0000", 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087768999999999998 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249433+0000", 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5333000000000004e-05 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249451+0000", 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8406e-05 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249656+0000", 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020463100000000001 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249685+0000", 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8943000000000001e-05 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264308+0000", 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014623486999999999 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264338+0000", 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0020999999999999e-05 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as2 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.928054+0000", 2024-09-15T02:49:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.807588521, 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0098530500000000003, 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928054+0000", 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928054+0000", 2024-09-15T02:49:21.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928056+0000", 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.097e-06 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928565+0000", 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000509626 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928642+0000", 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6363000000000001e-05 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928659+0000", 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7042999999999999e-05 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928798+0000", 2024-09-15T02:49:21.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000139405 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928820+0000", 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2235999999999999e-05 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937848+0000", 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0090280470000000008 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937907+0000", 2024-09-15T02:49:21.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8233000000000003e-05 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 62/33 ECSubWrite(tid=307, reqid=osd.13.0:307, at_version=62'49, trim_to=0'0, roll_forward_to=61'47))", 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.397908060000001, 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011170724999999999, 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.808 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337735+0000", 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0299999999999999e-07 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337799+0000", 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4206000000000002e-05 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338542+0000", 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074343499999999997 2024-09-15T02:49:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338565+0000", 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2931000000000001e-05 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338649+0000", 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4103000000000004e-05 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338667+0000", 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7419999999999999e-05 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.348862+0000", 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010195651 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.348905+0000", 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2376000000000002e-05 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 62/33 ECSubWrite(tid=310, reqid=osd.13.0:310, at_version=62'51, trim_to=0'0, roll_forward_to=61'47))", 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:06.339672+0000", 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.395969811000001, 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010374942, 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.339672+0000", 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.339672+0000", 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.339673+0000", 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2219999999999999e-06 2024-09-15T02:49:21.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.339714+0000", 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0840000000000002e-05 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.339772+0000", 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7655000000000003e-05 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.339783+0000", 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0976e-05 2024-09-15T02:49:21.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.340966+0000", 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001183239 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.340991+0000", 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5193000000000001e-05 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.350024+0000", 2024-09-15T02:49:21.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0090326780000000006 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.350047+0000", 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3139e-05 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:21.817 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.4489849100000001, 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020897651, 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286658+0000", 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.58e-07 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287075+0000", 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041690100000000001 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287318+0000", 2024-09-15T02:49:21.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024316499999999999 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287324+0000", 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9580000000000004e-06 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287521+0000", 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197746 2024-09-15T02:49:21.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287532+0000", 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366e-05 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307520+0000", 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019987953999999999 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307555+0000", 2024-09-15T02:49:21.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5102999999999999e-05 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.4485481580000004, 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020477789999999999, 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.822 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:21.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3200000000000001e-07 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287283+0000", 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018875200000000001 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287367+0000", 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4036000000000007e-05 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287374+0000", 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5340000000000002e-06 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295137+0000", 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077628599999999999 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295149+0000", 2024-09-15T02:49:21.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307560+0000", 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012411447000000001 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307572+0000", 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.826 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=413, reqid=client.4526.0:673, at_version=72'80, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.442705331, 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015088048, 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8799999999999999e-07 2024-09-15T02:49:21.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292992+0000", 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5479000000000002e-05 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293291+0000", 2024-09-15T02:49:21.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029850500000000002 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293295+0000", 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4179999999999999e-06 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302806+0000", 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095105680000000005 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302817+0000", 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1464999999999999e-05 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307993+0000", 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051757540000000003 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308025+0000", 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1671000000000002e-05 2024-09-15T02:49:21.831 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=414, reqid=client.4526.0:674, at_version=72'81, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.297747+0000", 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.4378951459999998, 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010873244000000001, 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297747+0000", 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297747+0000", 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297748+0000", 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.457e-06 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297774+0000", 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5737000000000001e-05 2024-09-15T02:49:21.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297839+0000", 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4671999999999993e-05 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297847+0000", 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6719999999999993e-06 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307606+0000", 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0097589809999999999 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307618+0000", 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1131000000000001e-05 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308556+0000", 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093819300000000003 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308620+0000", 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4400999999999995e-05 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.837 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:21.838 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:49:21.822+0000 7f3703cc0300 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:49:21.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:21.840 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:21.836+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:21.940 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:21.935+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:21.946 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:21.946 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:21.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.946 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:753 3.17s0 3.ca7af477 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.48450252799999999, 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48454223099999999, 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for peered", 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 753 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460754+0000", 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9709999999999998e-06 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460760+0000", 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7510000000000002e-06 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460773+0000", 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2136999999999999e-05 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460780+0000", 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471867+0000", 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011087001000000001 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:21.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471873+0000", 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000002e-06 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:754 3.17s0 3.ca7af477 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:21.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.48446694000000001, 2024-09-15T02:49:21.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.484617664, 2024-09-15T02:49:21.952 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.952 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for peered", 2024-09-15T02:49:21.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:21.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:21.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 754 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460788+0000", 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8899999999999999e-07 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460791+0000", 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.411e-06 2024-09-15T02:49:21.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460798+0000", 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4679999999999997e-06 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460802+0000", 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2889999999999999e-06 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471896+0000", 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011093591 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471897+0000", 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.331e-06 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.957 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:755 3.17s0 3.ca7af477 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:21.957 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:21.957 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.484447923, 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48469429200000003, 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for peered", 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 755 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.958 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460807+0000", 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460809+0000", 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460815+0000", 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1249999999999998e-06 2024-09-15T02:49:21.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460819+0000", 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9670000000000003e-06 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471918+0000", 2024-09-15T02:49:21.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098856000000001 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471919+0000", 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.962 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.963 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:766 3.17s0 3.228bc437 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:21.963 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:21.963 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.484430894, 2024-09-15T02:49:21.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48477028500000002, 2024-09-15T02:49:21.964 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.964 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for peered", 2024-09-15T02:49:21.964 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:21.964 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:21.964 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:21.964 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 766 2024-09-15T02:49:21.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.965 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:21.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:21.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460824+0000", 2024-09-15T02:49:21.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.27e-07 2024-09-15T02:49:21.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460831+0000", 2024-09-15T02:49:21.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-09-15T02:49:21.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460837+0000", 2024-09-15T02:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-09-15T02:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460841+0000", 2024-09-15T02:49:21.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5889999999999999e-06 2024-09-15T02:49:21.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471939+0000", 2024-09-15T02:49:21.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098591 2024-09-15T02:49:21.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:21.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471940+0000", 2024-09-15T02:49:21.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:21.973 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.973 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.973 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.973 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:767 3.17s0 3.228bc437 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:21.974 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:21.974 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.48440983599999998, 2024-09-15T02:49:21.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48484198499999998, 2024-09-15T02:49:21.974 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.974 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for peered", 2024-09-15T02:49:21.974 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:21.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:21.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:21.975 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 767 2024-09-15T02:49:21.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.975 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:21.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:21.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460845+0000", 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3200000000000001e-07 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460847+0000", 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.554e-06 2024-09-15T02:49:21.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460853+0000", 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020000000000002e-06 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460857+0000", 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0679999999999998e-06 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471961+0000", 2024-09-15T02:49:21.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011103755999999999 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471962+0000", 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3600000000000003e-07 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:768 3.17s0 3.228bc437 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.48439291800000001, 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48491570299999998, 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for peered", 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 768 2024-09-15T02:49:21.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460862+0000", 2024-09-15T02:49:21.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.34e-07 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460864+0000", 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6639999999999998e-06 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460870+0000", 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0859999999999998e-06 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460874+0000", 2024-09-15T02:49:21.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.563e-06 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471982+0000", 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011108122999999999 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471983+0000", 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6700000000000003e-07 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:21.984 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.985 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:21.985 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-15T02:49:21.985 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 6 2024-09-15T02:49:21.985 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:21.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:22.041 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:22.035+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:22.086 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:22.086 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:22.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:22.130 INFO:teuthology.orchestra.run.smithi152.stdout:3.es5 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.bs4 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.as5 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.5s6 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.8s6 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.18s1 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.7s1 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.18s6 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.17s1 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.15s6 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.15s1 2024-09-15T02:49:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:3.cs4 2024-09-15T02:49:22.132 INFO:teuthology.orchestra.run.smithi152.stdout:3.12s3 2024-09-15T02:49:22.132 INFO:teuthology.orchestra.run.smithi152.stdout:3.10s1 2024-09-15T02:49:22.132 INFO:teuthology.orchestra.run.smithi152.stdout:2.5 2024-09-15T02:49:22.132 INFO:teuthology.orchestra.run.smithi152.stdout:2.6 2024-09-15T02:49:22.132 INFO:teuthology.orchestra.run.smithi152.stdout:2.0 2024-09-15T02:49:22.141 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:22.136+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:22.198 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:22.198 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:22.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:22.198 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:22.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.198 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.829858911999999, 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.199 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:22.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:22.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.470287398000004, 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:22.207 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:22.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:22.213 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.861666933999999, 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026305674000000001, 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.214 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331938+0000", 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331943+0000", 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7559999999999999e-06 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331955+0000", 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331964+0000", 2024-09-15T02:49:22.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4069999999999993e-06 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343483+0000", 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011518995000000001 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343581+0000", 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8568999999999998e-05 2024-09-15T02:49:22.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.343713+0000", 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013195700000000001 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.348183+0000", 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044698639999999996 2024-09-15T02:49:22.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358029+0000", 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0098460219999999994 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358143+0000", 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114335 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358243+0000", 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9881000000000001e-05 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:22.221 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:22.222 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.861634256999999, 2024-09-15T02:49:22.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026584729000000001, 2024-09-15T02:49:22.222 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.222 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.222 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.222 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331971+0000", 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332135+0000", 2024-09-15T02:49:22.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000164098 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332158+0000", 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332165+0000", 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8669999999999996e-06 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344120+0000", 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011955222 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344205+0000", 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4895000000000006e-05 2024-09-15T02:49:22.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344336+0000", 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130844 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.349645+0000", 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053089770000000003 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358356+0000", 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087112079999999998 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358462+0000", 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105664 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358555+0000", 2024-09-15T02:49:22.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3110999999999997e-05 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.861424571999997, 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.229 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:49:22.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:49:22.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:49:22.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:49:22.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.237 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.624730954, 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.238 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:22.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:22.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.607212378, 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.793349951000003, 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:22.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:22.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.487106313000002, 2024-09-15T02:49:22.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.487048479999999, 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:22.265 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:22.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:22.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:22.272 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.487013294, 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:22.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.es1 49/33 ECSubWrite(tid=47, reqid=client.4526.0:403, at_version=49'26, trim_to=0'0, roll_forward_to=49'25))", 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.931374+0000", 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.262230108000001, 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020869234, 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931374+0000", 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931374+0000", 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931375+0000", 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0900000000000001e-07 2024-09-15T02:49:22.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931828+0000", 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045379699999999998 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931962+0000", 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013329999999999999 2024-09-15T02:49:22.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931969+0000", 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2609999999999996e-06 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932088+0000", 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118706 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932107+0000", 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9185999999999999e-05 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.952226+0000", 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020119103999999999 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.952244+0000", 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7671000000000001e-05 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.751708735000001, 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:22.284 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:22.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:22.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:22.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.289 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.913276522, 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:22.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:668 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283558+0000", 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.910047037, 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022510473999999999, 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 668 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.295 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283558+0000", 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283558+0000", 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283559+0000", 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5880000000000001e-06 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283565+0000", 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0270000000000001e-06 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283580+0000", 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4722999999999999e-05 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283590+0000", 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0113e-05 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283641+0000", 2024-09-15T02:49:22.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1372000000000002e-05 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283721+0000", 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9497000000000001e-05 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297157+0000", 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013436581 2024-09-15T02:49:22.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.297257+0000", 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9370999999999994e-05 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.306068+0000", 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0088112020000000006 2024-09-15T02:49:22.300 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:669 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283602+0000", 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.9100029909999998, 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025155818, 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.301 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 669 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283602+0000", 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283602+0000", 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283602+0000", 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283605+0000", 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.083e-06 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283612+0000", 2024-09-15T02:49:22.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4840000000000001e-06 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283616+0000", 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1030000000000004e-06 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283766+0000", 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015026899999999999 2024-09-15T02:49:22.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283926+0000", 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016048599999999999 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298126+0000", 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014199325 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298209+0000", 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3572999999999994e-05 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308757+0000", 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010548015000000001 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:670 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.9099826599999998, 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027236222000000001, 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 670 2024-09-15T02:49:22.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8800000000000003e-07 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283625+0000", 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3329999999999999e-06 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283630+0000", 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7450000000000001e-06 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283638+0000", 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7349999999999996e-06 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283962+0000", 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032339899999999998 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284074+0000", 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011208499999999999 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298432+0000", 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014357939 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298517+0000", 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310858+0000", 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012341388999999999 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.bs2 72/33 ECSubRead(tid=420, to_read={3:d2661d0e:::smithi19125844-43:head=0,4096,0}, subchunks={3:d2661d0e:::smithi19125844-43:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:22.314 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8828013209999996, 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085265647999999999, 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310804+0000", 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310814+0000", 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0115000000000001e-05 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395542+0000", 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.084728263999999998 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395552+0000", 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5899999999999997e-06 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395608+0000", 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5832999999999999e-05 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.396069+0000", 2024-09-15T02:49:22.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046113399999999998 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.fs3 74/53 ECSubWrite(tid=104, reqid=client.4526.0:696, at_version=74'58, trim_to=0'0, roll_forward_to=72'55))", 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.477755+0000", 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.7158498369999999, 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024386218000000001, 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.319 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.477755+0000", 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.477755+0000", 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.477755+0000", 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.08e-07 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.478363+0000", 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000608053 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.478608+0000", 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000245059 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.478618+0000", 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4860000000000003e-06 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.481747+0000", 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003129214 2024-09-15T02:49:22.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.481754+0000", 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8709999999999999e-06 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502007+0000", 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020253568 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502141+0000", 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000133559 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.fs3 74/53 ECSubWrite(tid=106, reqid=client.4526.0:711, at_version=74'60, trim_to=0'0, roll_forward_to=72'55))", 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.487321+0000", 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.7062833120000001, 2024-09-15T02:49:22.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022438979000000001, 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487321+0000", 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487321+0000", 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487322+0000", 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4300000000000002e-07 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487440+0000", 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118297 2024-09-15T02:49:22.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487629+0000", 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000189037 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487637+0000", 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.294e-06 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487729+0000", 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2383e-05 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.487738+0000", 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8969999999999997e-06 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.509733+0000", 2024-09-15T02:49:22.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021994565000000001 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.509760+0000", 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7662999999999999e-05 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.329 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:22.331 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:22.324+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:22.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:22.431 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:22.424+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:22.451 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:22.451 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:22.452 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:22.452 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:22.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:307: done (8 left) 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:308: rollback oid 45 current snap is 25 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 45 to 18 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:309: read oid 27 snap -1 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:309: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:310: snap_create 2024-09-15T02:49:22.468 INFO:tasks.rados.rados.0.smithi191.stdout:305: expect (ObjNum 93 snap 15 seq_num 93) 2024-09-15T02:49:22.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:49:22.533 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:22.527+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:302: finishing rollback tid 0 to smithi19125844-38 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:299: finishing rollback tid 1 to smithi19125844-47 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:299: finishing rollback tid 2 to smithi19125844-47 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 47 v 58 (ObjNum 66 snap 4 seq_num 66) dirty exists 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:302: finishing rollback tid 1 to smithi19125844-38 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:302: finishing rollback tid 2 to smithi19125844-38 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 38 v 50 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:300: finishing write tid 1 to smithi19125844-44 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:300: finishing write tid 2 to smithi19125844-44 2024-09-15T02:49:22.538 INFO:tasks.rados.rados.0.smithi191.stdout:300: finishing write tid 3 to smithi19125844-44 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:300: finishing write tid 4 to smithi19125844-44 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:300: finishing write tid 5 to smithi19125844-44 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:300: finishing write tid 6 to smithi19125844-44 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 44 v 48 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:300: left oid 44 (ObjNum 116 snap 23 seq_num 116) 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:308: finishing rollback tid 1 to smithi19125844-45 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 45 v 0 (ObjNum 33 snap 0 seq_num 565332800) dirty dne 2024-09-15T02:49:22.539 INFO:tasks.rados.rados.0.smithi191.stdout:309: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T02:49:22.559 INFO:tasks.rados.rados.0.smithi191.stdout:297: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-15T02:49:22.561 INFO:tasks.rados.rados.0.smithi191.stdout:301: expect (ObjNum 84 snap 9 seq_num 84) 2024-09-15T02:49:22.587 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:22.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:22.587 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:22.587 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:22.587 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:22.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:298: finishing copy_from racing read to smithi19125844-17 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:294: finishing write tid 1 to smithi19125844-48 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:294: finishing write tid 2 to smithi19125844-48 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:294: finishing write tid 3 to smithi19125844-48 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:294: finishing write tid 4 to smithi19125844-48 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:294: finishing write tid 5 to smithi19125844-48 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 44 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:294: left oid 48 (ObjNum 114 snap 23 seq_num 114) 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:298: finishing copy_from to smithi19125844-17 2024-09-15T02:49:22.634 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 17 v 25 (ObjNum 77 snap 8 seq_num 77) dirty exists 2024-09-15T02:49:22.635 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:22.628+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:22.669 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 3.as5 --file /var/log/ceph/exp.3.as5.6 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/33 ECSubWriteReply(tid=19, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.146807455999998, 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74030938599999996, 2024-09-15T02:49:22.689 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539611+0000", 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3680000000000001e-06 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539619+0000", 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278674+0000", 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73905563200000002 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278687+0000", 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.239e-05 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278784+0000", 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7149000000000001e-05 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.279919+0000", 2024-09-15T02:49:22.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001135645 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=24, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.099440166999997, 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30338358999999998, 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.693 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586978+0000", 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.51e-07 2024-09-15T02:49:22.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586981+0000", 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890125+0000", 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30314395700000002 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890138+0000", 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2627e-05 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890297+0000", 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015958899999999999 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890361+0000", 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3658999999999995e-05 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.696 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:284 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598091+0000", 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.088326377999998, 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019191204999999999, 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.697 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 284 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598091+0000", 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598091+0000", 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598095+0000", 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8199999999999998e-06 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598114+0000", 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9202999999999999e-05 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598163+0000", 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8420000000000001e-05 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598190+0000", 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7555999999999998e-05 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598409+0000", 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021893499999999999 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598431+0000", 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1843000000000001e-05 2024-09-15T02:49:22.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612108+0000", 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013677116 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.612283+0000", 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000175107 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617282+0000", 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0049992049999999996 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:285 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.088146303000002, 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019260777999999999, 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.702 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 285 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598273+0000", 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1670000000000002e-06 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598285+0000", 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1873e-05 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598308+0000", 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2895999999999999e-05 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598321+0000", 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3293e-05 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598586+0000", 2024-09-15T02:49:22.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026496499999999998 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598606+0000", 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9760000000000001e-05 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613160+0000", 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014553452 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613251+0000", 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1471000000000006e-05 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617532+0000", 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042809010000000002 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.707 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:286 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.087983274999999, 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019362717000000002, 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 286 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598437+0000", 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4940000000000002e-06 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598451+0000", 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.448e-05 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598503+0000", 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1983e-05 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598527+0000", 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4026e-05 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598660+0000", 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013317900000000001 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598681+0000", 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0296e-05 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613559+0000", 2024-09-15T02:49:22.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014878416 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613646+0000", 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6667000000000002e-05 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617797+0000", 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041511760000000003 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:414 3.3s0 3:cebed992:::smithi19125844-27:head [copy-from ver 4] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.430553733, 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034868557000000001, 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 414 2024-09-15T02:49:22.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:22.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255866+0000", 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.621e-06 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255879+0000", 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2738999999999999e-05 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255912+0000", 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3099000000000003e-05 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255927+0000", 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5013e-05 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256006+0000", 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8688000000000003e-05 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256016+0000", 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8379999999999997e-06 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262150+0000", 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061347040000000004 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262292+0000", 2024-09-15T02:49:22.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141595 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.275991+0000", 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013699266999999999 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282717+0000", 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067257910000000001 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290669+0000", 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079519480000000003 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290732+0000", 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3254000000000001e-05 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.719 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:415 3.3s0 3:cebed992:::smithi19125844-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.430466114000001, 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026655980999999999, 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 415 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255952+0000", 2024-09-15T02:49:22.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.108e-06 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255956+0000", 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.597e-06 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255965+0000", 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3789999999999995e-06 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255971+0000", 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3199999999999999e-06 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256090+0000", 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011949 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256109+0000", 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8644999999999999e-05 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262493+0000", 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063841519999999997 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262538+0000", 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4790999999999999e-05 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282336+0000", 2024-09-15T02:49:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019798651 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282486+0000", 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149037 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282607+0000", 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121811 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=77, last_complete=68'67, committed=1, applied=1))", 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.274056692, 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.106159008, 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412362+0000", 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9999999999999996e-07 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412370+0000", 2024-09-15T02:49:22.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4120000000000001e-06 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517947+0000", 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10557704800000001 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517957+0000", 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0173999999999999e-05 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518475+0000", 2024-09-15T02:49:22.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051777099999999999 2024-09-15T02:49:22.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518520+0000", 2024-09-15T02:49:22.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4703000000000002e-05 2024-09-15T02:49:22.732 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:636 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~606208] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.189401545000001, 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026984608, 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.733 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 636 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497019+0000", 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.066e-06 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497546+0000", 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052673800000000001 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497580+0000", 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4647999999999998e-05 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497592+0000", 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1669e-05 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497695+0000", 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103031 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497915+0000", 2024-09-15T02:49:22.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219607 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.499984+0000", 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020692800000000002 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518517+0000", 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018533353999999998 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523868+0000", 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053511779999999998 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524001+0000", 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132037 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:637 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~688128] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:22.738 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.181344322999999, 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022414014999999999, 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 637 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505076+0000", 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0570000000000001e-06 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505732+0000", 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065579899999999999 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505759+0000", 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6744999999999999e-05 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505786+0000", 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7518999999999999e-05 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.506354+0000", 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00056810900000000002 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.506548+0000", 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000193263 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508427+0000", 2024-09-15T02:49:22.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018793289999999999 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523607+0000", 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015180023000000001 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527407+0000", 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038003220000000001 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527487+0000", 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9849000000000001e-05 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:638 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~475136] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.178301945999999, 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.170696291, 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.744 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 638 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508118+0000", 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6110000000000001e-06 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508908+0000", 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00078945100000000004 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508947+0000", 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9298999999999998e-05 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508971+0000", 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4329e-05 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518663+0000", 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0096915070000000002 2024-09-15T02:49:22.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518760+0000", 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7740999999999998e-05 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.519373+0000", 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061209299999999997 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523608+0000", 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042356319999999996 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678746+0000", 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15513796099999999 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678812+0000", 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5667000000000007e-05 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.749 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:639 3.3s0 3:c4165a1c:::smithi19125844-22:head [setxattr _header (57) in=7b] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.177417423, 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.189219685, 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.750 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 639 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509002+0000", 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7579999999999999e-06 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509010+0000", 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3110000000000006e-06 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509031+0000", 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1234e-05 2024-09-15T02:49:22.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509052+0000", 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0962e-05 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522387+0000", 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013334693 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522514+0000", 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126723 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523057+0000", 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054339999999999998 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524433+0000", 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001375739 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698072+0000", 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17363879099999999 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698220+0000", 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014807399999999999 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:640 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.177330191999999, 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19189399200000001, 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 640 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509090+0000", 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2800000000000002e-06 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509095+0000", 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9679999999999997e-06 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509113+0000", 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8031000000000001e-05 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509122+0000", 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6509999999999996e-06 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524120+0000", 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014997610999999999 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524399+0000", 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027887000000000002 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698989+0000", 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.174589943 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.699138+0000", 2024-09-15T02:49:22.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149333 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.700981+0000", 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018433049999999999 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=80, last_complete=68'71, committed=1, applied=1))", 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.159203664, 2024-09-15T02:49:22.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15161024100000001, 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527215+0000", 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.088e-06 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527221+0000", 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9909999999999999e-06 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678531+0000", 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15131061000000001 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678545+0000", 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3597e-05 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678666+0000", 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012109599999999999 2024-09-15T02:49:22.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678824+0000", 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015785899999999999 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=81, last_complete=68'72, committed=1, applied=1))", 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.007941207, 2024-09-15T02:49:22.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020448245, 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:22.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678478+0000", 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6729999999999999e-06 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678489+0000", 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.059e-05 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697765+0000", 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0192768 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697777+0000", 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1749000000000001e-05 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697916+0000", 2024-09-15T02:49:22.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013848199999999999 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698925+0000", 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010089509999999999 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:699 3.12s0 3:4a71560d:::smithi19125844-16:head [rollback 24] snapc 26=[26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2176788529999998, 2024-09-15T02:49:22.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.064459370000000002, 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 699 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468741+0000", 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0420000000000001e-06 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468749+0000", 2024-09-15T02:49:22.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000002e-06 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468764+0000", 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5353999999999999e-05 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468773+0000", 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7649999999999999e-06 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495474+0000", 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026701740000000002 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495855+0000", 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038018500000000001 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512214+0000", 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016359598 2024-09-15T02:49:22.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512474+0000", 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025961499999999999 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.533198+0000", 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020724131 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:703 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2176370409999997, 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056538106999999997, 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 703 2024-09-15T02:49:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468781+0000", 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468786+0000", 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4909999999999999e-06 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468794+0000", 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0229999999999992e-06 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468803+0000", 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1910000000000006e-06 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.500871+0000", 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032068169000000001 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502024+0000", 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001152674 2024-09-15T02:49:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525089+0000", 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023065130999999999 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525319+0000", 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022968700000000001 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:704 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2176031060000003, 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056656818999999997, 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 704 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468815+0000", 2024-09-15T02:49:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4199999999999995e-07 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468819+0000", 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7359999999999999e-06 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468826+0000", 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0190000000000001e-06 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468829+0000", 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5499999999999999e-06 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502105+0000", 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033275546000000003 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502234+0000", 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012945 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525355+0000", 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023120971000000001 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525471+0000", 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011580499999999999 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:705 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2175834290000003, 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056845816, 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 705 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468838+0000", 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0690000000000002e-06 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468843+0000", 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7219999999999996e-06 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468849+0000", 2024-09-15T02:49:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0340000000000002e-06 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502534+0000", 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033684990999999997 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502672+0000", 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137701 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525531+0000", 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022859147 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525680+0000", 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014871700000000001 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(osd.12.22:3 3.12s0 3:48b768c1:::smithi19125844-48:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.515968+0000", 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.1704492699999998, 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019246940000000001, 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515968+0000", 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515968+0000", 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515970+0000", 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9350000000000001e-06 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515977+0000", 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5139999999999998e-06 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515995+0000", 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8425000000000001e-05 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.516003+0000", 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7959999999999996e-06 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.516081+0000", 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8308999999999999e-05 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.520820+0000", 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047385509999999997 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.535021+0000", 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014201346 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:22.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.535215+0000", 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000194064 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:22.794 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:22.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:22.797 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:22.787+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:22.896 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:22.894+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:22.902 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:22.903 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:22.903 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:22.903 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:22.913 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:22.997 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:22.992+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:23.059 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:23.059 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:23.059 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:23.059 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:23.059 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:23.068 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:23.098 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:23.092+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:871775f5:::smithi19125844-15:c, version: 48'19, data_included: [0~61440], data_size: 61440, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:871775f5:::smithi19125844-15:c@48'19, size: 245760, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:245760, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.172 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.725325494, 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80849678800000002, 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.174 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:23.174 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.174 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.175 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442896+0000", 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0449999999999999e-06 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442986+0000", 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9937000000000003e-05 2024-09-15T02:49:23.176 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.177 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.177 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.177 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.246998+0000", 2024-09-15T02:49:23.177 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80401223499999996 2024-09-15T02:49:23.177 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.177 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247018+0000", 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9735999999999999e-05 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247164+0000", 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000146093 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.251392+0000", 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042277419999999996 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:801a1da7:::smithi19125844-39:e, version: 49'21, data_included: [0~159744], data_size: 159744, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:e@49'21, size: 638976, copy_subset: [], clone_subset: {}, snapset: e=[]:{e=[e,b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:638976, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.902595439999999, 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010047222999999999, 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.179 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.180 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265626+0000", 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.325e-06 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265815+0000", 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018882999999999999 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265866+0000", 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0565000000000001e-05 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265876+0000", 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0273e-05 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.266085+0000", 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000209049 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.275672+0000", 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095871810000000002 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.182 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.883997476000001, 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087482150000000002, 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284497+0000", 2024-09-15T02:49:23.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027328299999999998 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284538+0000", 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1449999999999998e-05 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284547+0000", 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1020000000000003e-06 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.185 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284639+0000", 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1615999999999998e-05 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.292971+0000", 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083324509999999994 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.186 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [262144~221184], data_size: 221184, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1933312, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.879682104, 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014791191, 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:23.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288540+0000", 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9369999999999998e-06 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288962+0000", 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042141100000000001 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289058+0000", 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6339e-05 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289089+0000", 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0867000000000001e-05 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289255+0000", 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016596199999999999 2024-09-15T02:49:23.189 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.303330+0000", 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014074675 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:e, version: 49'26, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:e@49'26, size: 1081344, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.859218326000001, 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007188888, 2024-09-15T02:49:23.190 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309002+0000", 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309004+0000", 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4529999999999999e-06 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309408+0000", 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000404798 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309488+0000", 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9389999999999997e-05 2024-09-15T02:49:23.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309520+0000", 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2509999999999999e-05 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.309641+0000", 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120958 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.316191+0000", 2024-09-15T02:49:23.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0065497790000000004 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.843118449999999, 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012696989000000001, 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325103+0000", 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.333e-06 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325526+0000", 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000422788 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325605+0000", 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8932999999999998e-05 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325624+0000", 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8525999999999998e-05 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325780+0000", 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015595999999999999 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.337799+0000", 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012019449 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.197 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.82575336, 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01280455, 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:23.198 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342469+0000", 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6929999999999999e-06 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342850+0000", 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000381462 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342951+0000", 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000100559 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342971+0000", 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9720000000000001e-05 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.343066+0000", 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5088999999999997e-05 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.355272+0000", 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012206027 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.79805927, 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012408221000000001, 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:23.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370163+0000", 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370553+0000", 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038988100000000002 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370636+0000", 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2825999999999996e-05 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370655+0000", 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.916e-05 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370778+0000", 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012348700000000001 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.382570+0000", 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01179122 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.781943486999999, 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070382149999999996, 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.205 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386277+0000", 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386278+0000", 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2699999999999999e-06 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386706+0000", 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042724800000000003 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386783+0000", 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6976000000000003e-05 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.207 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386809+0000", 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6143e-05 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.386892+0000", 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3196999999999996e-05 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.393315+0000", 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0064233809999999997 2024-09-15T02:49:23.208 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.770967215999999, 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0076400929999999997, 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397255+0000", 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.576e-06 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397632+0000", 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000377265 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397715+0000", 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3213000000000004e-05 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397743+0000", 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7229999999999998e-05 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397885+0000", 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014198699999999999 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.404893+0000", 2024-09-15T02:49:23.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070088219999999996 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [786432~167936], data_size: 167936, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3817472, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.759204705999998, 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0053001360000000004, 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409016+0000", 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409017+0000", 2024-09-15T02:49:23.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.406e-06 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409301+0000", 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028334600000000002 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409374+0000", 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3652999999999994e-05 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409421+0000", 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6879000000000003e-05 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.409504+0000", 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2559999999999996e-05 2024-09-15T02:49:23.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.414316+0000", 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048122920000000001 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/53 ECSubWrite(tid=67, reqid=client.4526.0:522, at_version=60'31, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.900672852, 2024-09-15T02:49:23.217 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43332960599999998, 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:23.218 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8200000000000001e-07 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267569+0000", 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0713e-05 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699630+0000", 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43206165099999999 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699644+0000", 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3274e-05 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699794+0000", 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000150719 2024-09-15T02:49:23.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699816+0000", 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1895999999999998e-05 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700821+0000", 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010047210000000001 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700877+0000", 2024-09-15T02:49:23.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6249999999999998e-05 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 61/53 ECSubWrite(tid=102, reqid=osd.14.0:99, at_version=0'0, trim_to=0'0, roll_forward_to=61'43))", 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.81892384, 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0054209050000000002, 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.222 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349297+0000", 2024-09-15T02:49:23.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7499999999999999e-07 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349311+0000", 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3689999999999999e-05 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.224 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349385+0000", 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.3937999999999996e-05 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349392+0000", 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0790000000000003e-06 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349455+0000", 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2742000000000003e-05 2024-09-15T02:49:23.225 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.349465+0000", 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0259e-05 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354685+0000", 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052197889999999999 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:05.354718+0000", 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2632999999999997e-05 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=402, reqid=osd.13.0:402, at_version=72'63, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.258650+0000", 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.9095702010000002, 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060995349999999997, 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258650+0000", 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258650+0000", 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258651+0000", 2024-09-15T02:49:23.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.84e-07 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258674+0000", 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2929999999999999e-05 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258779+0000", 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010520500000000001 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.229 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258789+0000", 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0504e-05 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258848+0000", 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8316000000000002e-05 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258864+0000", 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6021000000000001e-05 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.230 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264720+0000", 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058567489999999996 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264750+0000", 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9526000000000002e-05 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=403, reqid=osd.13.0:403, at_version=72'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:23.231 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.909544414, 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0085463759999999996, 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8500000000000002e-07 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258694+0000", 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7482e-05 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258812+0000", 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000118245 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258817+0000", 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4379999999999997e-06 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259246+0000", 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042895599999999998 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.234 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259256+0000", 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0818e-05 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.267192+0000", 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0079355799999999994 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.267223+0000", 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0571999999999998e-05 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 72/53 ECSubWrite(tid=96, reqid=client.4526.0:667, at_version=0'0, trim_to=0'0, roll_forward_to=72'53))", 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.259587+0000", 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.9086331489999999, 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0052448989999999999, 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259587+0000", 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259587+0000", 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259588+0000", 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0759999999999999e-06 2024-09-15T02:49:23.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259602+0000", 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3128e-05 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259656+0000", 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4800999999999999e-05 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.238 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259665+0000", 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6999999999999997e-06 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259750+0000", 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4556999999999997e-05 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259761+0000", 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1652e-05 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264807+0000", 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050455769999999999 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.264832+0000", 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5408000000000001e-05 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.240 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.19s6 72/71 ECSubWrite(tid=132, reqid=osd.14.0:132, at_version=72'51, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.8924960439999996, 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0057855809999999997, 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275725+0000", 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9899999999999999e-07 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275754+0000", 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8892999999999999e-05 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275812+0000", 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8069000000000003e-05 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275821+0000", 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5420000000000005e-06 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275876+0000", 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5547999999999997e-05 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.275893+0000", 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6863000000000001e-05 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.281483+0000", 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0055895529999999997 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.281510+0000", 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7214000000000001e-05 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=409, reqid=osd.13.0:409, at_version=72'67, trim_to=0'0, roll_forward_to=72'65))", 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.8782257290000004, 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0050522639999999999, 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.245 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.289995+0000", 2024-09-15T02:49:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4200000000000002e-07 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290019+0000", 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3717e-05 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290089+0000", 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0205999999999998e-05 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.247 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290095+0000", 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9560000000000002e-06 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290171+0000", 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6384000000000006e-05 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.290185+0000", 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3219e-05 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.248 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.295015+0000", 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0048300770000000003 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.295047+0000", 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2462999999999997e-05 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.249 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.19s6 72/71 ECSubWrite(tid=134, reqid=osd.14.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=72'51))", 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.291564+0000", 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.876656112, 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.006517892, 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291564+0000", 2024-09-15T02:49:23.250 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291564+0000", 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291566+0000", 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.215e-06 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.251 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291583+0000", 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7119e-05 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291640+0000", 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7188000000000002e-05 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291651+0000", 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1365e-05 2024-09-15T02:49:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291705+0000", 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3922000000000003e-05 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.291716+0000", 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0862999999999999e-05 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.298052+0000", 2024-09-15T02:49:23.253 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0063362660000000001 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.298082+0000", 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9954000000000002e-05 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs3 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:23.254 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.671052749, 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021673943000000001, 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:23.255 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8700000000000002e-07 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499075+0000", 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001907176 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.256 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499139+0000", 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3250000000000006e-05 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499151+0000", 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2371000000000001e-05 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499260+0000", 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000109064 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.257 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499277+0000", 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6659000000000001e-05 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.518812+0000", 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019535858 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:23.258 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.518842+0000", 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9278e-05 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:23.259 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:23.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:23.261 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:23.256+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:23.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:23.361 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:23.356+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:23.398 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:49:23.399 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-15T02:49:23.399 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-15T02:49:23.399 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:49:23.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:23.462 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:23.456+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:294: done (10 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:297: done (9 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:298: done (8 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:299: done (7 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:300: done (6 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:301: done (5 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:302: done (4 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:305: done (3 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:308: done (2 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:309: done (1 left) 2024-09-15T02:49:23.470 INFO:tasks.rados.rados.0.smithi191.stdout:310: done (0 left) 2024-09-15T02:49:23.471 INFO:tasks.rados.rados.0.smithi191.stdout:311: setattr oid 44 current snap is 26 2024-09-15T02:49:23.472 INFO:tasks.rados.rados.0.smithi191.stdout:312: rollback oid 20 current snap is 26 2024-09-15T02:49:23.472 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 20 to 25 2024-09-15T02:49:23.472 INFO:tasks.rados.rados.0.smithi191.stdout:313: rmattr oid 48 current snap is 26 2024-09-15T02:49:23.472 INFO:tasks.rados.rados.0.smithi191.stdout:313: done (2 left) 2024-09-15T02:49:23.472 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 23 current snap is 26 2024-09-15T02:49:23.472 INFO:tasks.rados.rados.0.smithi191.stdout:314: seq_num 118 ranges {0=442368,442368=753664,1196032=655360,1851392=147456} 2024-09-15T02:49:23.472 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:49:23.483 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:23.484 INFO:tasks.rados.rados.0.smithi191.stdout:314: writing smithi19125844-23 from 0 to 442368 tid 1 2024-09-15T02:49:23.488 INFO:tasks.rados.rados.0.smithi191.stdout:314: writing smithi19125844-23 from 442368 to 1196032 tid 2 2024-09-15T02:49:23.492 INFO:tasks.rados.rados.0.smithi191.stdout:314: writing smithi19125844-23 from 1196032 to 1851392 tid 3 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:314: writing smithi19125844-23 from 1851392 to 1998848 tid 4 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:315: read oid 36 snap -1 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:315: expect (ObjNum 74 snap 7 seq_num 74) 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:316: delete oid 32 current snap is 26 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:312: finishing rollback tid 0 to smithi19125844-20 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:312: finishing rollback tid 1 to smithi19125844-20 2024-09-15T02:49:23.494 INFO:tasks.rados.rados.0.smithi191.stdout:312: finishing rollback tid 2 to smithi19125844-20 2024-09-15T02:49:23.495 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 20 v 51 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T02:49:23.495 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 44 v 50 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-15T02:49:23.507 INFO:tasks.rados.rados.0.smithi191.stdout:315: expect (ObjNum 74 snap 7 seq_num 74) 2024-09-15T02:49:23.515 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:23.516 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:23.516 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:23.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:23.541 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:23.563 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:23.558+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:23.566 INFO:tasks.rados.rados.0.smithi191.stdout:314: finishing write tid 1 to smithi19125844-23 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: finishing write tid 2 to smithi19125844-23 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: finishing write tid 3 to smithi19125844-23 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: finishing write tid 4 to smithi19125844-23 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: finishing write tid 5 to smithi19125844-23 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: finishing write tid 6 to smithi19125844-23 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 23 v 88 (ObjNum 118 snap 26 seq_num 118) dirty exists 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: left oid 23 (ObjNum 118 snap 26 seq_num 118) 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:311: done (4 left) 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:312: done (3 left) 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:314: done (2 left) 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:315: done (1 left) 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:316: done (0 left) 2024-09-15T02:49:23.567 INFO:tasks.rados.rados.0.smithi191.stdout:317: snap_remove snap 22 2024-09-15T02:49:23.568 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:23.593 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:23.621 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:23.621 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:23.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:23.653 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:23.695 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:23.695 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:23.687+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:23.734 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:23.764 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:23.762+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:23.766 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:23.766 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:23.768 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:23.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:23.798 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:23.826 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:23.853 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:23.865 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:23.860+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:23.883 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:23.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:23.883+0000 7fe902ffd700 0 -- 172.21.15.6:0/1307924065 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fe9041926b0 msgr2=0x7fe904194a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:23.897 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:23.897 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:23.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:23.919 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:23.919 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:23.919 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:23.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:23.949 INFO:teuthology.orchestra.run.smithi152.stderr:Exporting 3.as5 info 3.as5( v 71'56 (0'0,71'56] local-lis/les=33/34 n=6 ec=33/33 lis/c=33/33 les/c/f=34/34/0 sis=33) 2024-09-15T02:49:23.958 INFO:teuthology.orchestra.run.smithi152.stderr:Read 5#3:517ce0e6:::smithi19125844-47:13# 2024-09-15T02:49:23.959 INFO:teuthology.orchestra.run.smithi152.stderr:Read 5#3:517ce0e6:::smithi19125844-47:head# 2024-09-15T02:49:23.960 INFO:teuthology.orchestra.run.smithi152.stderr:Read 5#3:54c1cad6:::smithi19125844-28:e# 2024-09-15T02:49:23.960 INFO:teuthology.orchestra.run.smithi152.stderr:Read 5#3:54c1cad6:::smithi19125844-28:1d# 2024-09-15T02:49:23.964 INFO:teuthology.orchestra.run.smithi152.stderr:Read 5#3:54c1cad6:::smithi19125844-28:head# 2024-09-15T02:49:23.965 INFO:teuthology.orchestra.run.smithi152.stderr:Read 5#3:5c026bb9:::smithi19125844-49:head# 2024-09-15T02:49:23.965 INFO:teuthology.orchestra.run.smithi152.stderr:Export successful 2024-09-15T02:49:23.966 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:23.964+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:24.018 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.018 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:24.067 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:24.060+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:24.129 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.129 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:24.169 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:24.163+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:24.236 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.237 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:24.270 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:24.264+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:24.350 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.351 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:24.373 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:24.370+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:24.461 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.461 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:24.474 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:24.467+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:24.479 INFO:tasks.rados.rados.0.smithi191.stdout:317: done (0 left) 2024-09-15T02:49:24.479 INFO:tasks.rados.rados.0.smithi191.stdout:318: copy_from oid 10 from oid 48 current snap is 26 2024-09-15T02:49:24.479 INFO:tasks.rados.rados.0.smithi191.stdout:319: snap_create 2024-09-15T02:49:24.517 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 3.as5 2024-09-15T02:49:24.527 INFO:tasks.rados.rados.0.smithi191.stdout:318: finishing copy_from racing read to smithi19125844-10 2024-09-15T02:49:24.530 INFO:tasks.rados.rados.0.smithi191.stdout:318: finishing copy_from to smithi19125844-10 2024-09-15T02:49:24.530 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 10 v 44 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-15T02:49:24.568 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.568 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:24.578 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:24.571+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:24.674 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.674 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.678 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:24.676+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:24.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:24.780 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:24.772+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:24.791 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.791 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:24.880 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:24.875+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:24.902 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:24.903 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:24.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:24.981 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:24.976+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:25.012 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.012 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:25.081 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:25.076+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:25.167 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.167 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:25.182 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:25.178+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:25.273 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.274 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:25.283 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:25.275+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:25.383 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:25.376+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:25.384 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.384 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:318: done (1 left) 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:319: done (0 left) 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:320: read oid 47 snap -1 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:320: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:321: copy_from oid 36 from oid 31 current snap is 27 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:322: rollback oid 12 current snap is 27 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 12 to 25 2024-09-15T02:49:25.484 INFO:tasks.rados.rados.0.smithi191.stdout:323: delete oid 41 current snap is 27 2024-09-15T02:49:25.485 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:25.476+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:25.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:49:25.494 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.494 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.501 INFO:tasks.rados.rados.0.smithi191.stdout:322: finishing rollback tid 0 to smithi19125844-12 2024-09-15T02:49:25.502 INFO:tasks.rados.rados.0.smithi191.stdout:323: done (3 left) 2024-09-15T02:49:25.502 INFO:tasks.rados.rados.0.smithi191.stdout:324: snap_create 2024-09-15T02:49:25.502 INFO:tasks.rados.rados.0.smithi191.stdout:322: finishing rollback tid 1 to smithi19125844-12 2024-09-15T02:49:25.505 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:25.585 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:25.582+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:25.603 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.603 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.611 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:25.686 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:25.684+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:25.707 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.708 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:25.742 INFO:teuthology.orchestra.run.smithi152.stdout: marking collection for removal 2024-09-15T02:49:25.742 INFO:teuthology.orchestra.run.smithi152.stdout:setting '_remove' omap key 2024-09-15T02:49:25.742 INFO:teuthology.orchestra.run.smithi152.stdout:finish_remove_pgs 3.as5_head removing 3.as5 2024-09-15T02:49:25.745 INFO:teuthology.orchestra.run.smithi152.stdout:Remove successful 2024-09-15T02:49:25.788 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:25.784+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:25.818 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.818 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:25.889 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:25.886+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:25.900 INFO:tasks.rados.rados.0.smithi191.stdout:322: finishing rollback tid 2 to smithi19125844-12 2024-09-15T02:49:25.900 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 12 v 92 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T02:49:25.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:25.929 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:25.929 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:25.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:25.941 INFO:tasks.rados.rados.0.smithi191.stdout:321: finishing copy_from racing read to smithi19125844-36 2024-09-15T02:49:25.990 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:25.983+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:26.035 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:26.036 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:26.045 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:26.075 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:26.091 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:26.083+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:26.110 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:26.139 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:26.155 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:26.155 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:26.155 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:26.155 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:26.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:26.169 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:26.192 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:26.188+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:26.194 INFO:tasks.rados.rados.0.smithi191.stdout:320: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:49:26.197 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:26.225 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:26.252 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:26.268 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:26.268 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:26.268 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:26.268 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:26.268 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:26.275 INFO:tasks.rados.rados.0.smithi191.stdout:321: finishing copy_from to smithi19125844-36 2024-09-15T02:49:26.275 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 36 v 68 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-09-15T02:49:26.277 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:26.279 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:26.283 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.as5.6 2024-09-15T02:49:26.293 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:26.290+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:26.306 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:26.333 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:26.362 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:26.375 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:26.375 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 61.635424526000001, 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:49:26.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:403 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~606208] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.614685831999999, 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.195887642, 2024-09-15T02:49:26.380 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 403 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757022+0000", 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.593e-06 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757800+0000", 2024-09-15T02:49:26.382 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077824999999999997 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757830+0000", 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0057e-05 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757844+0000", 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3947999999999999e-05 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.383 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757925+0000", 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0249999999999999e-05 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.758082+0000", 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015777599999999999 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759386+0000", 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001303217 2024-09-15T02:49:26.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777348+0000", 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017962568000000002 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952681+0000", 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17533264000000001 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952907+0000", 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000226343 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:404 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~720896] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.610467227999997, 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.192160255, 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.386 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 404 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.387 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761280+0000", 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2138999999999998e-05 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762400+0000", 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001119451 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762433+0000", 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3238000000000003e-05 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762446+0000", 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3308e-05 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768118+0000", 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056721360000000004 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768285+0000", 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000166482 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769662+0000", 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013768140000000001 2024-09-15T02:49:26.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780475+0000", 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010813578000000001 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953275+0000", 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17280025199999999 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953398+0000", 2024-09-15T02:49:26.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012285700000000001 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:405 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~49152] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.609236490000001, 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19129260000000001, 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.392 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 405 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762470+0000", 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4699999999999999e-06 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762527+0000", 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7151999999999998e-05 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762550+0000", 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2311999999999999e-05 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762559+0000", 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777614+0000", 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015055628999999999 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777696+0000", 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1840000000000002e-05 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778120+0000", 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042372900000000002 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780527+0000", 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00240712 2024-09-15T02:49:26.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953622+0000", 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17309514500000001 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953761+0000", 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013926300000000001 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:406 3.es0 3:71c1a91c:::smithi19125844-45:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.609136530000001, 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19225968700000001, 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 406 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762570+0000", 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.034e-06 2024-09-15T02:49:26.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762575+0000", 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4079999999999997e-06 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762592+0000", 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.715e-05 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762600+0000", 2024-09-15T02:49:26.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6480000000000003e-06 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779660+0000", 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017060035000000001 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779757+0000", 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6537000000000001e-05 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780137+0000", 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038070299999999998 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781414+0000", 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012764230000000001 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954649+0000", 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17323481900000001 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954828+0000", 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017992999999999999 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:407 3.es0 3:71c1a91c:::smithi19125844-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.609096657999999, 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19532122900000001, 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.403 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 407 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762610+0000", 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762614+0000", 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0659999999999997e-06 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762626+0000", 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2490000000000001e-05 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762639+0000", 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2704000000000001e-05 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781806+0000", 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019166823999999999 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.782082+0000", 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027635000000000002 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955680+0000", 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17359843999999999 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955807+0000", 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012658299999999999 2024-09-15T02:49:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.957930+0000", 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021228499999999999 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.13.22:5 3.es0 3:740f3d5b:::smithi19125844-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.596565677999997, 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.177222558, 2024-09-15T02:49:26.408 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775141+0000", 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775146+0000", 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5630000000000002e-06 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775159+0000", 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3509999999999999e-05 2024-09-15T02:49:26.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775165+0000", 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8130000000000001e-06 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778738+0000", 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003572867 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778948+0000", 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00021028299999999999 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952362+0000", 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17341393199999999 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s5 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:26.412 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.260527+0000", 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.111178649000003, 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021017170000000002, 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260527+0000", 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.413 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260527+0000", 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260528+0000", 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.55e-06 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260685+0000", 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000156646 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260761+0000", 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5753000000000005e-05 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.260774+0000", 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3132999999999999e-05 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.264984+0000", 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042104040000000001 2024-09-15T02:49:26.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.265024+0000", 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0046999999999998e-05 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281494+0000", 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016469609 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281544+0000", 2024-09-15T02:49:26.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0028000000000003e-05 2024-09-15T02:49:26.417 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.417 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.417 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.417 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:529 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~802816] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:26.420 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:26.420 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.116880220999999, 2024-09-15T02:49:26.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.030230086, 2024-09-15T02:49:26.420 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.420 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 529 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254827+0000", 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6390000000000001e-06 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256167+0000", 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013402900000000001 2024-09-15T02:49:26.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256196+0000", 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8762999999999999e-05 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256203+0000", 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6229999999999997e-06 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036663860000000002 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.260012+0000", 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014185099999999999 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017728349999999999 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277478+0000", 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015693650999999999 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285001+0000", 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075225120000000003 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285055+0000", 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4536000000000001e-05 2024-09-15T02:49:26.425 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:530 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~638976] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.112959066999998, 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026488489, 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.426 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 530 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258748+0000", 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.66e-06 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259495+0000", 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00074681400000000005 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259522+0000", 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7552999999999999e-05 2024-09-15T02:49:26.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259538+0000", 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6220000000000001e-05 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269841+0000", 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010302477000000001 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269927+0000", 2024-09-15T02:49:26.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5993000000000002e-05 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.270621+0000", 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00069455099999999996 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277769+0000", 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0071479769999999998 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.430 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285179+0000", 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074099320000000002 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285235+0000", 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5312000000000002e-05 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.431 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:531 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~131072] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259559+0000", 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.112145879, 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025800705, 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 531 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259559+0000", 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259559+0000", 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259560+0000", 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3900000000000003e-07 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259669+0000", 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010854 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259678+0000", 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9250000000000001e-06 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259685+0000", 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7800000000000003e-06 2024-09-15T02:49:26.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.275061+0000", 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015376413 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.275145+0000", 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4052000000000001e-05 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.275597+0000", 2024-09-15T02:49:26.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00045223799999999999 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277773+0000", 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021752310000000001 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.436 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285318+0000", 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075454190000000003 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285360+0000", 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2168e-05 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:532 3.es0 3:7f1f21a6:::smithi19125844-10:head [setxattr _header (57) in=7b] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259688+0000", 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.112016880999999, 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026024156, 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 532 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259688+0000", 2024-09-15T02:49:26.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259688+0000", 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259689+0000", 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2e-07 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259691+0000", 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2199999999999999e-06 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259695+0000", 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.856e-06 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259702+0000", 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9240000000000001e-06 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.276644+0000", 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016942352000000001 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.276725+0000", 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1017e-05 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277096+0000", 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00037033699999999999 2024-09-15T02:49:26.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277952+0000", 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00085680499999999998 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285648+0000", 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0076958 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285713+0000", 2024-09-15T02:49:26.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.4325000000000005e-05 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:533 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.112000373000001, 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027459596999999999, 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.443 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 533 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.89e-07 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259707+0000", 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5460000000000001e-06 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259711+0000", 2024-09-15T02:49:26.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1439999999999996e-06 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259715+0000", 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3340000000000003e-06 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277889+0000", 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018173363000000001 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277948+0000", 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8913999999999999e-05 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286109+0000", 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0081616199999999996 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286156+0000", 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6928000000000002e-05 2024-09-15T02:49:26.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.287165+0000", 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010083589999999999 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.3s4 68/33 ECSubWrite(tid=78, reqid=client.4526.0:636, at_version=68'69, trim_to=0'0, roll_forward_to=68'67))", 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.499687+0000", 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 13.872018536000001, 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.023067649999999999, 2024-09-15T02:49:26.448 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.499687+0000", 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.499687+0000", 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.499689+0000", 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9740000000000001e-06 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.500254+0000", 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00056510100000000004 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.500330+0000", 2024-09-15T02:49:26.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6292999999999996e-05 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.500343+0000", 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2527000000000001e-05 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.500495+0000", 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015266499999999999 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.500516+0000", 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1143000000000001e-05 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.522664+0000", 2024-09-15T02:49:26.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022147310999999999 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.522754+0000", 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0636000000000004e-05 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s2 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:26.453 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.8376042430000004, 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027643175999999998, 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.454 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534111+0000", 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8090000000000004e-06 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534425+0000", 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031419599999999998 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542230+0000", 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0078045069999999996 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542239+0000", 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542375+0000", 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000135897 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542396+0000", 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1237999999999999e-05 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561688+0000", 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019291689000000001 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561744+0000", 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6178999999999999e-05 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.457 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s2 77/76 ECSubWrite(tid=251, reqid=client.4526.0:761, at_version=77'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.541892+0000", 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.829813433, 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019899546000000001, 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.541892+0000", 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.541892+0000", 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.541893+0000", 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2799999999999995e-07 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542207+0000", 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031445000000000003 2024-09-15T02:49:26.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.546788+0000", 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045805680000000001 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.546797+0000", 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.460 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552614+0000", 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0058175190000000002 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552623+0000", 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3839999999999997e-06 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561758+0000", 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0091356119999999996 2024-09-15T02:49:26.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561791+0000", 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3244999999999998e-05 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.8s2 77/76 ECSubRead(tid=258, to_read={3:173bfaf3:::smithi19125844-44:18=262144,262144,0}, subchunks={3:173bfaf3:::smithi19125844-44:18=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.7958454850000001, 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119859681, 2024-09-15T02:49:26.462 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.463 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575861+0000", 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0640000000000001e-06 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575876+0000", 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5165e-05 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.464 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694960+0000", 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119084367 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694980+0000", 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9930000000000001e-05 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695069+0000", 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8737000000000001e-05 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695719+0000", 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065041800000000005 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=112, attrs_read=0))", 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.7913554520000003, 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13363014300000001, 2024-09-15T02:49:26.466 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580351+0000", 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2419999999999999e-06 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580721+0000", 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00037001299999999998 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712609+0000", 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13188792199999999 2024-09-15T02:49:26.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712623+0000", 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.429e-05 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712782+0000", 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000158791 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.469 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.713980+0000", 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011978850000000001 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.6243282069999996, 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21641586800000001, 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747379+0000", 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:26.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747806+0000", 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042687399999999999 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962958+0000", 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21515266799999999 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962974+0000", 2024-09-15T02:49:26.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6237999999999999e-05 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963059+0000", 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4968000000000004e-05 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963793+0000", 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00073351999999999998 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.473 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.12s5 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 3.8498631620000001, 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028389666000000001, 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.474 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521843+0000", 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.122e-06 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522442+0000", 2024-09-15T02:49:26.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00059853399999999996 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533671+0000", 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011229075 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533686+0000", 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5265999999999999e-05 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533798+0000", 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000112198 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533821+0000", 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2229999999999999e-05 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550165+0000", 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016344826 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550232+0000", 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6414999999999993e-05 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.478 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:26.480 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:26.480 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:26.480 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:26.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:26.471+0000 7fb09f7fe700 0 -- 172.21.15.6:0/3502352917 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fb0a0192750 msgr2=0x7fb0a0194b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:26.481 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:26.471+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:26.481 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:26.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:26.501 INFO:tasks.rados.rados.0.smithi191.stdout:320: done (3 left) 2024-09-15T02:49:26.501 INFO:tasks.rados.rados.0.smithi191.stdout:321: done (2 left) 2024-09-15T02:49:26.501 INFO:tasks.rados.rados.0.smithi191.stdout:322: done (1 left) 2024-09-15T02:49:26.501 INFO:tasks.rados.rados.0.smithi191.stdout:324: done (0 left) 2024-09-15T02:49:26.501 INFO:tasks.rados.rados.0.smithi191.stdout:325: snap_create 2024-09-15T02:49:26.503 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:49:26.580 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:26.576+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:26.633 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:26.633 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:26.633 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:26.634 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:26.644 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:26.682 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:26.680+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:26.747 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:26.747 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:26.747 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:26.747 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:26.748 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:26.759 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:26.808 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:26.806+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:26.862 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:26.862 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:26.862 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.238950582000001, 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:26.863 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.864 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:49:26.865 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.866 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.866 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:49:26.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:49:26.866 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.866 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.867 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.867 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:49:26.867 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:49:26.867 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.867 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.867 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:416 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~638976] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.596742093000003, 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026381341999999999, 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.868 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 416 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261066+0000", 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1159999999999999e-06 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262194+0000", 2024-09-15T02:49:26.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001127783 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262245+0000", 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0880000000000001e-05 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262260+0000", 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5065999999999999e-05 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.871 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262382+0000", 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012151599999999999 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262403+0000", 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1344000000000001e-05 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266706+0000", 2024-09-15T02:49:26.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004303355 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266794+0000", 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7717000000000001e-05 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268260+0000", 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001466406 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285160+0000", 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016899773 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287372+0000", 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022116140000000002 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287445+0000", 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2772000000000001e-05 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:417 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.587348143, 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028569959999999998, 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.875 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 417 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270459+0000", 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.035e-06 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271813+0000", 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001354037 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271834+0000", 2024-09-15T02:49:26.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0302e-05 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271844+0000", 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9070000000000002e-06 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274939+0000", 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003095169 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275041+0000", 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000102045 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276123+0000", 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010819880000000001 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285162+0000", 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009039432 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.298984+0000", 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013821599 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299027+0000", 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3445999999999997e-05 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:418 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~147456] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.585945641999999, 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027304571999999999, 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 418 2024-09-15T02:49:26.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271861+0000", 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1829999999999999e-06 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272037+0000", 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017563799999999999 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272050+0000", 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3506000000000001e-05 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272057+0000", 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999998e-06 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281725+0000", 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096680910000000002 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281887+0000", 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016169599999999999 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282772+0000", 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088466100000000002 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285439+0000", 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026669340000000001 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299113+0000", 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013674009000000001 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299164+0000", 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1573999999999998e-05 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.887 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:419 3.0s0 3:038350ba:::smithi19125844-11:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.585740344999998, 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.031729234000000002, 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 419 2024-09-15T02:49:26.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.889 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272066+0000", 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9000000000000005e-07 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272070+0000", 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1740000000000002e-06 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272078+0000", 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5329999999999997e-06 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272083+0000", 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7859999999999996e-06 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284520+0000", 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012437190000000001 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284611+0000", 2024-09-15T02:49:26.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0816000000000006e-05 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284926+0000", 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031568500000000001 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287991+0000", 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030649599999999998 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303704+0000", 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015713142999999999 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303794+0000", 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9957000000000006e-05 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.893 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:420 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.585717338000002, 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033404771999999999, 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 420 2024-09-15T02:49:26.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272089+0000", 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.51e-07 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272091+0000", 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1730000000000002e-06 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272096+0000", 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.904e-06 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272099+0000", 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2059999999999999e-06 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285286+0000", 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013187488000000001 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285374+0000", 2024-09-15T02:49:26.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7392999999999995e-05 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304237+0000", 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018863347999999999 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304386+0000", 2024-09-15T02:49:26.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014841 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.305493+0000", 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001107299 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:429 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:26.899 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.585643990999998, 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022788739999999998, 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 429 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272162+0000", 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:26.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272166+0000", 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9269999999999998e-06 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272174+0000", 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6569999999999997e-06 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272178+0000", 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6260000000000003e-06 2024-09-15T02:49:26.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274127+0000", 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001949295 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274178+0000", 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0547999999999998e-05 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.903 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.294950+0000", 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020772112999999998 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 50/33 ECSubReadReply(tid=60, attrs_read=0))", 2024-09-15T02:49:26.904 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.580370723000001, 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017580023, 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0100000000000004e-07 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293753+0000", 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016317505999999999 2024-09-15T02:49:26.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293859+0000", 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105707 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293862+0000", 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4819999999999999e-06 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293910+0000", 2024-09-15T02:49:26.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7763000000000003e-05 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.295015+0000", 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011048640000000001 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:457 3.9s0 3:9c49cf3d:::smithi19125844-14:head [append 0~0] snapc 11=[11,10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 30.610266927000001, 2024-09-15T02:49:26.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017317169, 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 457 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247540+0000", 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1629999999999999e-06 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.910 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247546+0000", 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9320000000000001e-06 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247563+0000", 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7268000000000001e-05 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247573+0000", 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8970000000000003e-06 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247766+0000", 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019318499999999999 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247773+0000", 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8600000000000004e-06 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254774+0000", 2024-09-15T02:49:26.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070010150000000002 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254891+0000", 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000117342 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.255604+0000", 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071258400000000005 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.256332+0000", 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072847400000000001 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264776+0000", 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0084437450000000008 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264856+0000", 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9703999999999994e-05 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.914 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.12.22:2 3.9s0 3:94f52c52:::smithi19125844-20:head [assert-version v18,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 27.529949072000001, 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59485876100000001, 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327859+0000", 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.446e-06 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327869+0000", 2024-09-15T02:49:26.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7629999999999995e-06 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327898+0000", 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9054e-05 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327913+0000", 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4990999999999999e-05 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328005+0000", 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2015000000000006e-05 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328180+0000", 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017567599999999999 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.922715+0000", 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59453481600000002 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.918 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:630 3.9s0 3:9c49cf3d:::smithi19125844-14:head [copy-from ver 31] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.463894509999999, 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021438329999999998, 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 630 2024-09-15T02:49:26.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:26.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393915+0000", 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7570000000000002e-06 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393951+0000", 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6347999999999998e-05 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393987+0000", 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6405999999999997e-05 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394003+0000", 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5956000000000001e-05 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394131+0000", 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012749300000000001 2024-09-15T02:49:26.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394155+0000", 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3754999999999999e-05 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401719+0000", 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075645340000000004 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401977+0000", 2024-09-15T02:49:26.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000257543 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409969+0000", 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007991873 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414563+0000", 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045945830000000002 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:26.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415199+0000", 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063606199999999996 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415349+0000", 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015002 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:631 3.9s0 3:9c49cf3d:::smithi19125844-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:26.925 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.463753572, 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017944699000000001, 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 631 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394053+0000", 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3179999999999999e-06 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394059+0000", 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9120000000000003e-06 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394075+0000", 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5809999999999999e-05 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394085+0000", 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0586e-05 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394258+0000", 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017270999999999999 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394277+0000", 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9236999999999999e-05 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402333+0000", 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0080553299999999994 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402447+0000", 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00011408600000000001 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411771+0000", 2024-09-15T02:49:26.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093238820000000004 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411860+0000", 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8857999999999995e-05 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411997+0000", 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013697000000000001 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.931 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 72/71 ECSubWriteReply(tid=134, last_complete=72'51, committed=1, applied=1))", 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.564963539000001, 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060209891000000001, 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292843+0000", 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3099999999999996e-07 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292848+0000", 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6729999999999997e-06 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352898+0000", 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060049335000000002 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352910+0000", 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.256e-05 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352988+0000", 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7831999999999996e-05 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.353052+0000", 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3559999999999995e-05 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.935 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:688 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 8.4189325260000007, 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037318727000000003, 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 688 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438875+0000", 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5809999999999999e-06 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438889+0000", 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3444999999999999e-05 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438927+0000", 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7842000000000001e-05 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438940+0000", 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3473e-05 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439086+0000", 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014553999999999999 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439109+0000", 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3283000000000001e-05 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448063+0000", 2024-09-15T02:49:26.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089537490000000004 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448274+0000", 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000210984 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476192+0000", 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027917830000000001 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:689 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 8.4188435389999992, 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037724385999999999, 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.941 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 689 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438963+0000", 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5300000000000002e-07 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438968+0000", 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9180000000000002e-06 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438976+0000", 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6470000000000004e-06 2024-09-15T02:49:26.943 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438981+0000", 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9130000000000002e-06 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439306+0000", 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032486700000000001 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439319+0000", 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2442999999999999e-05 2024-09-15T02:49:26.944 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448872+0000", 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095532159999999998 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449323+0000", 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00045103599999999998 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476686+0000", 2024-09-15T02:49:26.945 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027363393 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:690 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 8.4187613890000002, 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038145563, 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:26.946 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 690 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.947 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439045+0000", 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2200000000000003e-07 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439048+0000", 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.76e-06 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439064+0000", 2024-09-15T02:49:26.948 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5336e-05 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439069+0000", 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8849999999999998e-06 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439382+0000", 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031364799999999998 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:26.949 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439397+0000", 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4661000000000001e-05 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449673+0000", 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010276298 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449816+0000", 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014270799999999999 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.950 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.477190+0000", 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027373544999999999 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDScrubReserve(3.19s4 RELEASE e75)", 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 6.5757704989999999, 2024-09-15T02:49:26.951 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98297010699999998, 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.952 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282036+0000", 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2270000000000001e-06 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282044+0000", 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6720000000000004e-06 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264750+0000", 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98270659599999999 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.953 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264763+0000", 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2466000000000001e-05 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264912+0000", 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000149524 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264954+0000", 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1461000000000002e-05 2024-09-15T02:49:26.954 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.265005+0000", 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1161e-05 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s4 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.3234929720000004, 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018239887999999999, 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.956 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534314+0000", 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7549999999999999e-06 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534957+0000", 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00064235200000000001 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535041+0000", 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4050999999999999e-05 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.957 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535056+0000", 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5588999999999999e-05 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535177+0000", 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120925 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535207+0000", 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9864000000000001e-05 2024-09-15T02:49:26.958 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552499+0000", 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017291890000000001 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552552+0000", 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3461999999999997e-05 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.959 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.bs4 77/76 [PushOp(3:d394ce14:::smithi19125844-2:head, version: 71'77, data_included: [262144~241664], data_size: 241664, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d394ce14:::smithi19125844-2:head@71'77, size: 2015232, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2015232, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 5.2213595579999996, 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.24636945099999999, 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.960 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636447+0000", 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.637e-06 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.961 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636697+0000", 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024920600000000003 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867157+0000", 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.23046036 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867176+0000", 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.929e-05 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.962 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867332+0000", 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015594099999999999 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.882815+0000", 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015483017 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:26.963 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.3603189960000002, 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025018460999999999, 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.964 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497487+0000", 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8000000000000002e-07 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499654+0000", 2024-09-15T02:49:26.965 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00216738 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499711+0000", 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7398999999999997e-05 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499723+0000", 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2019e-05 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499830+0000", 2024-09-15T02:49:26.966 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000106718 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499850+0000", 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9732e-05 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522437+0000", 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022587494 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:26.967 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522505+0000", 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7439000000000003e-05 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:26.968 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:26.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:26.970 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:26.968+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:27.070 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:27.068+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:27.078 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.079 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:27.079 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:27.079 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:27.171 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:27.168+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:27.191 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.191 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:27.191 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:27.191 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:27.191 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:27.272 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:27.270+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:27.301 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.301 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:27.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:27.301 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:27.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 43.728340082000003, 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:49:27.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:49:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 43.718397776000003, 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099046280000000004, 2024-09-15T02:49:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579386+0000", 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.536e-06 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579616+0000", 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230366 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579660+0000", 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4489e-05 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579668+0000", 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4800000000000004e-06 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579731+0000", 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2733999999999999e-05 2024-09-15T02:49:27.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579742+0000", 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589278+0000", 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095358219999999994 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589289+0000", 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0596e-05 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 36.647393399999999, 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015141226000000001, 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.311 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650389+0000", 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650600+0000", 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021155999999999999 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650679+0000", 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8641999999999997e-05 2024-09-15T02:49:27.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650693+0000", 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4294000000000001e-05 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650801+0000", 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108132 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650828+0000", 2024-09-15T02:49:27.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6565999999999999e-05 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665503+0000", 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014675142 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665530+0000", 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6509999999999999e-05 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.605299166000002, 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011398508, 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.22e-07 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693204+0000", 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072156 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693258+0000", 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3134999999999999e-05 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693266+0000", 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5259999999999993e-06 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693333+0000", 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6580000000000003e-05 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693344+0000", 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0971999999999999e-05 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703854+0000", 2024-09-15T02:49:27.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010510265 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703881+0000", 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7248000000000001e-05 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.694524+0000", 2024-09-15T02:49:27.320 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.603257677000002, 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091159489999999999, 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694524+0000", 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694524+0000", 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694525+0000", 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8699999999999995e-07 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695089+0000", 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00056474900000000002 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695146+0000", 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6920999999999999e-05 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695157+0000", 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0828e-05 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695241+0000", 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3974999999999995e-05 2024-09-15T02:49:27.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695254+0000", 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2405999999999999e-05 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703597+0000", 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083438360000000003 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703640+0000", 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2647e-05 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=161, reqid=client.4526.0:365, at_version=49'24, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.702878+0000", 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.594904178999997, 2024-09-15T02:49:27.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0090111389999999996, 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702878+0000", 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702878+0000", 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702879+0000", 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1599999999999999e-06 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.702998+0000", 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118973 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703046+0000", 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8607999999999999e-05 2024-09-15T02:49:27.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703058+0000", 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1574e-05 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703916+0000", 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00085808100000000001 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703927+0000", 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0798000000000001e-05 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.711856+0000", 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079289029999999993 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.711889+0000", 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3042e-05 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.331 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.043468537000003, 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016572159999999999, 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254314+0000", 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1100000000000002e-07 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255086+0000", 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077269100000000005 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255147+0000", 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1005999999999998e-05 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255166+0000", 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8581999999999999e-05 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255267+0000", 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100973 2024-09-15T02:49:27.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255289+0000", 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2565000000000002e-05 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270855+0000", 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015565733999999999 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270885+0000", 2024-09-15T02:49:27.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0298e-05 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=191, reqid=client.4526.0:410, at_version=50'33, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.261882+0000", 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.035899401000002, 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0094284799999999995, 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261882+0000", 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261882+0000", 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261883+0000", 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9600000000000008e-07 2024-09-15T02:49:27.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.261996+0000", 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113041 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262066+0000", 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9851999999999995e-05 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262078+0000", 2024-09-15T02:49:27.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1591e-05 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.265641+0000", 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003562843 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.265656+0000", 2024-09-15T02:49:27.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5531000000000001e-05 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.271284+0000", 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056282629999999997 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.271311+0000", 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6363000000000002e-05 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.028977582000003, 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018797873, 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8999999999999998e-07 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269160+0000", 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035577800000000001 2024-09-15T02:49:27.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3349000000000001e-05 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269211+0000", 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9439999999999999e-06 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269283+0000", 2024-09-15T02:49:27.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2263999999999998e-05 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269296+0000", 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3145e-05 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287519+0000", 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018222955999999998 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287602+0000", 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3146999999999995e-05 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.020704956000003, 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010563233999999999, 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277078+0000", 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1500000000000004e-07 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278860+0000", 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017825339999999999 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278958+0000", 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7705000000000001e-05 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278971+0000", 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3481e-05 2024-09-15T02:49:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279037+0000", 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5848999999999999e-05 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279045+0000", 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6359999999999994e-06 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287611+0000", 2024-09-15T02:49:27.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085663470000000002 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287640+0000", 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8966999999999999e-05 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.369970511999998, 2024-09-15T02:49:27.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010079322999999999, 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927813+0000", 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9630000000000001e-06 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928385+0000", 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057185600000000004 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928464+0000", 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8615999999999994e-05 2024-09-15T02:49:27.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928480+0000", 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6696000000000001e-05 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928602+0000", 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121209 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928622+0000", 2024-09-15T02:49:27.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.052e-05 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937813+0000", 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091904280000000005 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937891+0000", 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8034999999999994e-05 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.353 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(3.1s2 RELEASE e76)", 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.6281470430000002, 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43008043499999998, 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669637+0000", 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.917e-06 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669648+0000", 2024-09-15T02:49:27.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1653e-05 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099516+0000", 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42986811200000002 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099534+0000", 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7997000000000001e-05 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099653+0000", 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011822100000000001 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099681+0000", 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8496999999999998e-05 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099715+0000", 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4038000000000002e-05 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8140560590000003, 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012988501, 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:27.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483727+0000", 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2500000000000004e-07 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483886+0000", 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000159028 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483939+0000", 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3640999999999997e-05 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483950+0000", 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0462000000000001e-05 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.484771+0000", 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082086099999999999 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.496714+0000", 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011943684 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.361 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.7962477349999997, 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010341001000000001, 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501535+0000", 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2309999999999999e-06 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501651+0000", 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115538 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501708+0000", 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7479000000000003e-05 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501723+0000", 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4815e-05 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501805+0000", 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2000000000000001e-05 2024-09-15T02:49:27.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511875+0000", 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010069938 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:127fa005:::smithi19125844-18:head, version: 36'10, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:127fa005:::smithi19125844-18:head@36'10, size: 1212416, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.78798777, 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012919893999999999, 2024-09-15T02:49:27.365 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509795+0000", 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8599999999999996e-07 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510077+0000", 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028238699999999999 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510142+0000", 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4518999999999997e-05 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510161+0000", 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8751999999999999e-05 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511930+0000", 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017696719999999999 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.522714+0000", 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010783578 2024-09-15T02:49:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.18s0 77/76 ECSubWriteReply(tid=15, last_complete=77'49, committed=1, applied=1))", 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.515275+0000", 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.7825065599999999, 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093799920000000002, 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.515275+0000", 2024-09-15T02:49:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.515275+0000", 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.515276+0000", 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4600000000000003e-07 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.515281+0000", 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2580000000000003e-06 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.515329+0000", 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7175999999999997e-05 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.515337+0000", 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1480000000000006e-06 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.523152+0000", 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078152499999999993 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.524655+0000", 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015032139999999999 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:27.372 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.7632132739999999, 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0218164, 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:27.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6199999999999996e-07 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534737+0000", 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016761400000000001 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534787+0000", 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9836000000000002e-05 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534795+0000", 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1550000000000007e-06 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534859+0000", 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4162e-05 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534875+0000", 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6305000000000001e-05 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556345+0000", 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021469637999999999 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556385+0000", 2024-09-15T02:49:27.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9827999999999999e-05 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=251, reqid=client.4526.0:761, at_version=77'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.755418508, 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014142234, 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542364+0000", 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8700000000000005e-07 2024-09-15T02:49:27.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542468+0000", 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010437099999999999 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542522+0000", 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3295999999999999e-05 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542529+0000", 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1709999999999996e-06 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544564+0000", 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00203476 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544579+0000", 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5605000000000001e-05 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556487+0000", 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011907687 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556506+0000", 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556999999999999e-05 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.381 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:18, version: 70'39, data_included: [262144~253952], data_size: 253952, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:18@70'39, size: 2064384, copy_subset: [], clone_subset: {}, snapset: 27=[]:{e=[b],18=[18],27=[27,26,25,24,21,1d]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2064384, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.599559621, 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010750387, 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698224+0000", 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2570000000000002e-06 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698584+0000", 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035969000000000002 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698700+0000", 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116018 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698723+0000", 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3173e-05 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698830+0000", 2024-09-15T02:49:27.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010714599999999999 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.708973+0000", 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010142103 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 81/76 ECSubRead(tid=114, to_read={3:517ce0e6:::smithi19125844-47:head=0,475136,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.8086721130000001, 2024-09-15T02:49:27.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69833002899999996, 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489111+0000", 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7269999999999999e-06 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489134+0000", 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3033000000000001e-05 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187052+0000", 2024-09-15T02:49:27.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69791777600000005 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187066+0000", 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3567000000000001e-05 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187169+0000", 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010303599999999999 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187440+0000", 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027088999999999998 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.389 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.390 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:27.386+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:27.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:27.433 INFO:teuthology.orchestra.run.smithi152.stdout:get_pg_num_history pg_num_history pg_num_history(e74 pg_nums {1={23=1},2={26=8},3={33=16,71=26}} deleted_pools ) 2024-09-15T02:49:27.433 INFO:teuthology.orchestra.run.smithi152.stdout:Importing pgid 3.as5 2024-09-15T02:49:27.433 INFO:teuthology.orchestra.run.smithi152.stdout:pool 3 pg_num 26 -> 26 2024-09-15T02:49:27.434 INFO:teuthology.orchestra.run.smithi152.stdout:Write 5#3:517ce0e6:::smithi19125844-47:13# 2024-09-15T02:49:27.451 INFO:teuthology.orchestra.run.smithi152.stdout:Write 5#3:517ce0e6:::smithi19125844-47:head# 2024-09-15T02:49:27.452 INFO:teuthology.orchestra.run.smithi152.stdout:snapset 13=[]:{13=[b]} 2024-09-15T02:49:27.452 INFO:teuthology.orchestra.run.smithi152.stdout:Write 5#3:54c1cad6:::smithi19125844-28:e# 2024-09-15T02:49:27.458 INFO:teuthology.orchestra.run.smithi152.stdout:Write 5#3:54c1cad6:::smithi19125844-28:1d# 2024-09-15T02:49:27.471 INFO:teuthology.orchestra.run.smithi152.stdout:Write 5#3:54c1cad6:::smithi19125844-28:head# 2024-09-15T02:49:27.471 INFO:teuthology.orchestra.run.smithi152.stdout:snapset 1d=[]:{e=[b],1d=[1d,18]} 2024-09-15T02:49:27.475 INFO:teuthology.orchestra.run.smithi152.stdout:Write 5#3:5c026bb9:::smithi19125844-49:head# 2024-09-15T02:49:27.476 INFO:teuthology.orchestra.run.smithi152.stdout:snapset 0=[]:{} 2024-09-15T02:49:27.481 INFO:teuthology.orchestra.run.smithi152.stdout:write_pg epoch 71 info 3.as5( v 71'56 (0'0,71'56] local-lis/les=33/34 n=6 ec=33/33 lis/c=33/33 les/c/f=34/34/0 sis=33) 2024-09-15T02:49:27.482 INFO:teuthology.orchestra.run.smithi152.stdout:Import successful 2024-09-15T02:49:27.491 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:27.490+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:27.496 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.496 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:27.496 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:27.496 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:27.516 INFO:tasks.rados.rados.0.smithi191.stdout:325: done (0 left) 2024-09-15T02:49:27.516 INFO:tasks.rados.rados.0.smithi191.stdout:326: rollback oid 29 current snap is 29 2024-09-15T02:49:27.516 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 29 to 19 2024-09-15T02:49:27.516 INFO:tasks.rados.rados.0.smithi191.stdout:327: read oid 34 snap -1 2024-09-15T02:49:27.517 INFO:tasks.rados.rados.0.smithi191.stdout:327: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-15T02:49:27.517 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 8 current snap is 29 2024-09-15T02:49:27.517 INFO:tasks.rados.rados.0.smithi191.stdout:328: seq_num 121 ranges {1933312=491520,2424832=802816,3227648=720896,3948544=32768} 2024-09-15T02:49:27.537 INFO:tasks.rados.rados.0.smithi191.stdout:328: writing smithi19125844-8 from 1933312 to 2424832 tid 1 2024-09-15T02:49:27.542 INFO:tasks.rados.rados.0.smithi191.stdout:328: writing smithi19125844-8 from 2424832 to 3227648 tid 2 2024-09-15T02:49:27.546 INFO:tasks.rados.rados.0.smithi191.stdout:328: writing smithi19125844-8 from 3227648 to 3948544 tid 3 2024-09-15T02:49:27.547 INFO:tasks.rados.rados.0.smithi191.stdout:328: writing smithi19125844-8 from 3948544 to 3981312 tid 4 2024-09-15T02:49:27.547 INFO:tasks.rados.rados.0.smithi191.stdout:329: delete oid 45 current snap is 29 2024-09-15T02:49:27.547 INFO:tasks.rados.rados.0.smithi191.stdout:326: finishing rollback tid 1 to smithi19125844-29 2024-09-15T02:49:27.547 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 29 v 39 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-15T02:49:27.547 INFO:tasks.rados.rados.0.smithi191.stdout:327: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:328: finishing write tid 1 to smithi19125844-8 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:328: finishing write tid 2 to smithi19125844-8 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:328: finishing write tid 3 to smithi19125844-8 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:328: finishing write tid 4 to smithi19125844-8 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:326: done (3 left) 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:327: done (2 left) 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:329: done (1 left) 2024-09-15T02:49:27.574 INFO:tasks.rados.rados.0.smithi191.stdout:330: rollback oid 48 current snap is 29 2024-09-15T02:49:27.575 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 48 to 17 2024-09-15T02:49:27.575 INFO:tasks.rados.rados.0.smithi191.stdout:331: setattr oid 22 current snap is 29 2024-09-15T02:49:27.578 INFO:tasks.rados.rados.0.smithi191.stdout:330: finishing rollback tid 0 to smithi19125844-48 2024-09-15T02:49:27.578 INFO:tasks.rados.rados.0.smithi191.stdout:332: read oid 13 snap -1 2024-09-15T02:49:27.578 INFO:tasks.rados.rados.0.smithi191.stdout:332: expect deleted 2024-09-15T02:49:27.578 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 37 current snap is 29 2024-09-15T02:49:27.578 INFO:tasks.rados.rados.0.smithi191.stdout:333: seq_num 123 ranges {2293760=524288,2818048=573440,3391488=688128,4079616=294912} 2024-09-15T02:49:27.593 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:27.590+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:27.595 INFO:tasks.rados.rados.0.smithi191.stdout:333: writing smithi19125844-37 from 2293760 to 2818048 tid 1 2024-09-15T02:49:27.599 INFO:tasks.rados.rados.0.smithi191.stdout:333: writing smithi19125844-37 from 2818048 to 3391488 tid 2 2024-09-15T02:49:27.604 INFO:tasks.rados.rados.0.smithi191.stdout:333: writing smithi19125844-37 from 3391488 to 4079616 tid 3 2024-09-15T02:49:27.606 INFO:tasks.rados.rados.0.smithi191.stdout:333: writing smithi19125844-37 from 4079616 to 4374528 tid 4 2024-09-15T02:49:27.606 INFO:tasks.rados.rados.0.smithi191.stdout:334: snap_create 2024-09-15T02:49:27.606 INFO:tasks.rados.rados.0.smithi191.stdout:330: finishing rollback tid 1 to smithi19125844-48 2024-09-15T02:49:27.606 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 48 v 47 (ObjNum 564587184 snap 21890 seq_num 304) dirty dne 2024-09-15T02:49:27.606 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 22 v 94 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-15T02:49:27.622 INFO:tasks.rados.rados.0.smithi191.stdout:333: finishing write tid 1 to smithi19125844-37 2024-09-15T02:49:27.624 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.624 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:27.624 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:27.624 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:27.624 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.632 INFO:tasks.rados.rados.0.smithi191.stdout:333: finishing write tid 2 to smithi19125844-37 2024-09-15T02:49:27.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:27.641 INFO:tasks.rados.rados.0.smithi191.stdout:333: finishing write tid 3 to smithi19125844-37 2024-09-15T02:49:27.642 INFO:tasks.rados.rados.0.smithi191.stdout:333: finishing write tid 4 to smithi19125844-37 2024-09-15T02:49:27.643 INFO:tasks.rados.rados.0.smithi191.stdout:333: finishing write tid 5 to smithi19125844-37 2024-09-15T02:49:27.645 INFO:tasks.rados.rados.0.smithi191.stdout:333: finishing write tid 6 to smithi19125844-37 2024-09-15T02:49:27.645 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 37 v 99 (ObjNum 123 snap 29 seq_num 123) dirty exists 2024-09-15T02:49:27.646 INFO:tasks.rados.rados.0.smithi191.stdout:333: left oid 37 (ObjNum 123 snap 29 seq_num 123) 2024-09-15T02:49:27.670 INFO:tasks.rados.rados.0.smithi191.stdout:328: finishing write tid 5 to smithi19125844-8 2024-09-15T02:49:27.675 INFO:tasks.rados.rados.0.smithi191.stdout:328: finishing write tid 6 to smithi19125844-8 2024-09-15T02:49:27.675 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 8 v 108 (ObjNum 121 snap 29 seq_num 121) dirty exists 2024-09-15T02:49:27.675 INFO:tasks.rados.rados.0.smithi191.stdout:328: left oid 8 (ObjNum 121 snap 29 seq_num 121) 2024-09-15T02:49:27.694 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:27.692+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:27.728 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.728 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:27.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:27.728 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:27.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.728 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.154742812000002, 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:49:27.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:49:27.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.142226626999999, 2024-09-15T02:49:27.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:49:27.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:49:27.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.141187467000002, 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:49:27.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:49:27.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:49:27.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 37.073436461, 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015031548, 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651048+0000", 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4900000000000001e-07 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651266+0000", 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000218233 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651324+0000", 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8544000000000001e-05 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651335+0000", 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336000000000001e-05 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651408+0000", 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3415000000000001e-05 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651424+0000", 2024-09-15T02:49:27.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6001000000000001e-05 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666045+0000", 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014620861000000001 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666079+0000", 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3809000000000001e-05 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.032064077000001, 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015570855999999999, 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692736+0000", 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031561799999999999 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692809+0000", 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3174000000000001e-05 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692821+0000", 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1627000000000001e-05 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692906+0000", 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5172000000000004e-05 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692920+0000", 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4361e-05 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707954+0000", 2024-09-15T02:49:27.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015033451999999999 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707991+0000", 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6714000000000001e-05 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as2 49/33 ECSubWrite(tid=31, reqid=client.4526.0:360, at_version=49'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:27.751 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.029762873000003, 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011075066, 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.694721+0000", 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9400000000000001e-07 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695264+0000", 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054240199999999999 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695308+0000", 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4595999999999998e-05 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695316+0000", 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.605e-06 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695420+0000", 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000104505 2024-09-15T02:49:27.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.695441+0000", 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.084e-05 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705756+0000", 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010315094 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.705796+0000", 2024-09-15T02:49:27.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9629999999999998e-05 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:27.756 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.975001921999997, 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014530774999999999, 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749483+0000", 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4799999999999997e-07 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749717+0000", 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000234408 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749768+0000", 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1155000000000001e-05 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749777+0000", 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1609999999999992e-06 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749853+0000", 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5803000000000006e-05 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749872+0000", 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556e-05 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.763971+0000", 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014098934 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.764013+0000", 2024-09-15T02:49:27.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1810000000000001e-05 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.470121495999997, 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012056918, 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254363+0000", 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7199999999999997e-07 2024-09-15T02:49:27.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254896+0000", 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053267899999999999 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255034+0000", 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013821100000000001 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255066+0000", 2024-09-15T02:49:27.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255135+0000", 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8885000000000002e-05 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255148+0000", 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2498e-05 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266383+0000", 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011235187000000001 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266419+0000", 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6358e-05 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.433346890999999, 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011905962000000001, 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291138+0000", 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291342+0000", 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203403 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291572+0000", 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230084 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291579+0000", 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6990000000000003e-06 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291641+0000", 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2093000000000003e-05 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291650+0000", 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303018+0000", 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011368296 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303043+0000", 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4901000000000001e-05 2024-09-15T02:49:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.433112500999997, 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011692335, 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291548+0000", 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000177021 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291673+0000", 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012469299999999999 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291678+0000", 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296939+0000", 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052609120000000004 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296948+0000", 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303051+0000", 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006102433 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303064+0000", 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2948999999999999e-05 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/33 ECSubWrite(tid=212, reqid=client.4526.0:450, at_version=52'22, trim_to=0'0, roll_forward_to=49'21))", 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 31.476024318, 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015878961, 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248460+0000", 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249338+0000", 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087768999999999998 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249433+0000", 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5333000000000004e-05 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249451+0000", 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8406e-05 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249656+0000", 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020463100000000001 2024-09-15T02:49:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249685+0000", 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8943000000000001e-05 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264308+0000", 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014623486999999999 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264338+0000", 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0020999999999999e-05 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 62/33 ECSubWrite(tid=307, reqid=osd.13.0:307, at_version=62'49, trim_to=0'0, roll_forward_to=61'47))", 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.386749722000001, 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011170724999999999, 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337735+0000", 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0299999999999999e-07 2024-09-15T02:49:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337799+0000", 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4206000000000002e-05 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338542+0000", 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074343499999999997 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338565+0000", 2024-09-15T02:49:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2931000000000001e-05 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338649+0000", 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4103000000000004e-05 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338667+0000", 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7419999999999999e-05 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.348862+0000", 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010195651 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.348905+0000", 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2376000000000002e-05 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.437826572000001, 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020897651, 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286658+0000", 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.58e-07 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287075+0000", 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041690100000000001 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287318+0000", 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024316499999999999 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287324+0000", 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9580000000000004e-06 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287521+0000", 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197746 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287532+0000", 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366e-05 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307520+0000", 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019987953999999999 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307555+0000", 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5102999999999999e-05 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.43738982, 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020477789999999999, 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3200000000000001e-07 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287283+0000", 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018875200000000001 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287367+0000", 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4036000000000007e-05 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287374+0000", 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5340000000000002e-06 2024-09-15T02:49:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295137+0000", 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077628599999999999 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295149+0000", 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307560+0000", 2024-09-15T02:49:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012411447000000001 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307572+0000", 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=413, reqid=client.4526.0:673, at_version=72'80, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.431546993, 2024-09-15T02:49:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015088048, 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8799999999999999e-07 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292992+0000", 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5479000000000002e-05 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293291+0000", 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029850500000000002 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293295+0000", 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4179999999999999e-06 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302806+0000", 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095105680000000005 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302817+0000", 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1464999999999999e-05 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307993+0000", 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051757540000000003 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308025+0000", 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1671000000000002e-05 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:c, version: 71'82, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:c@71'82, size: 1081344, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.0097983020000001, 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013023088, 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714687+0000", 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.866e-06 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715107+0000", 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419293 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715195+0000", 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8758999999999995e-05 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715215+0000", 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0143e-05 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715370+0000", 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000154611 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.727709+0000", 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012338416 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:head, version: 71'83, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:head@71'83, size: 2293760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.7389614120000001, 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011824942, 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985524+0000", 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.533e-06 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985735+0000", 2024-09-15T02:49:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021152599999999999 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985816+0000", 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0516e-05 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985833+0000", 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7413000000000001e-05 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985974+0000", 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141161 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.997347+0000", 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011372793000000001 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7f1f21a6:::smithi19125844-10:head, version: 71'85, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:7f1f21a6:::smithi19125844-10:head@71'85, size: 1736704, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:27.804 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.4798648639999996, 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014391049, 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244620+0000", 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4300000000000001e-06 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245318+0000", 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069816099999999999 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245404+0000", 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5782e-05 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245423+0000", 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8272999999999999e-05 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245544+0000", 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121785 2024-09-15T02:49:27.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.259010+0000", 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013465618 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=472, reqid=client.4526.0:793, at_version=79'84, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.241016675, 2024-09-15T02:49:27.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013978862999999999, 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483468+0000", 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483595+0000", 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012675800000000001 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483644+0000", 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9200000000000003e-05 2024-09-15T02:49:27.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483653+0000", 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4030000000000007e-06 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483747+0000", 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4239000000000004e-05 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483762+0000", 2024-09-15T02:49:27.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5698000000000001e-05 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497313+0000", 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013550859 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497446+0000", 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013259400000000001 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=473, reqid=client.4526.0:794, at_version=79'85, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2348014459999996, 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014861180999999999, 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489684+0000", 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.282e-06 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490105+0000", 2024-09-15T02:49:27.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042170800000000002 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490163+0000", 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8158000000000001e-05 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490173+0000", 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4350000000000003e-06 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:27.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490301+0000", 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127928 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490314+0000", 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3104e-05 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504495+0000", 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014181545 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:27.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504543+0000", 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8021e-05 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:27.817 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:27.819 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:27.816+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:27.919 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:27.911+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:27.967 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:27.967 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:27.967 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:27.967 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:27.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:28.020 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:28.015+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:28.028 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /var/log/ceph/exp.3.as5.6 2024-09-15T02:49:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:28.071 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:28.071 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:28.071 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:28.071 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:28.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:28.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:49:28.121 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:28.118+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:28.177 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 63.809699492, 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:28.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:28.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:28.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 63.450127977999998, 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.185 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:28.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.191 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 62.841474837, 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026584729000000001, 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331971+0000", 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:49:28.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332135+0000", 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000164098 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332158+0000", 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332165+0000", 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8669999999999996e-06 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344120+0000", 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011955222 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344205+0000", 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4895000000000006e-05 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344336+0000", 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130844 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.349645+0000", 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053089770000000003 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358356+0000", 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087112079999999998 2024-09-15T02:49:28.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358462+0000", 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105664 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358555+0000", 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3110999999999997e-05 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 62.841265151999998, 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.198 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:49:28.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:49:28.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:28.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:28.203 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 62.604571534000002, 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:28.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:28.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 62.587052958000001, 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:28.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.213 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 60.773190530999997, 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:28.214 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:28.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:28.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:28.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:28.220 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.466946892999999, 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.221 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:28.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.226 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.46688906, 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.227 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:28.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:28.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:28.232 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.466853874000002, 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:28.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:28.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.731549314999999, 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:28.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.893117102000001, 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.242 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:28.245 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:670 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.88982324, 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027236222000000001, 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.246 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 670 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8800000000000003e-07 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283625+0000", 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3329999999999999e-06 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283630+0000", 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7450000000000001e-06 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283638+0000", 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7349999999999996e-06 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283962+0000", 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032339899999999998 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284074+0000", 2024-09-15T02:49:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011208499999999999 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298432+0000", 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014357939 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298517+0000", 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310858+0000", 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012341388999999999 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.bs2 72/33 ECSubRead(tid=420, to_read={3:d2661d0e:::smithi19125844-43:head=0,4096,0}, subchunks={3:d2661d0e:::smithi19125844-43:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.862641901, 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085265647999999999, 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.251 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310804+0000", 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310814+0000", 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0115000000000001e-05 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395542+0000", 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.084728263999999998 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395552+0000", 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5899999999999997e-06 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395608+0000", 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5832999999999999e-05 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.396069+0000", 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046113399999999998 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.254 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:753 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.7126934279999997, 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027324345, 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 753 2024-09-15T02:49:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460754+0000", 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9709999999999998e-06 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460760+0000", 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7510000000000002e-06 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460773+0000", 2024-09-15T02:49:28.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2136999999999999e-05 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460780+0000", 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471867+0000", 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011087001000000001 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471873+0000", 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000002e-06 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.484980+0000", 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0131073370000001 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485369+0000", 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038873500000000002 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488076+0000", 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027074099999999999 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:754 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.7126578400000003, 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0276021209999999, 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.260 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 754 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460788+0000", 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8899999999999999e-07 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460791+0000", 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.411e-06 2024-09-15T02:49:28.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460798+0000", 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4679999999999997e-06 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460802+0000", 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2889999999999999e-06 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471896+0000", 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011093591 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471897+0000", 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.331e-06 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485874+0000", 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0139766539999999 2024-09-15T02:49:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485962+0000", 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8481999999999998e-05 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488389+0000", 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024273060000000002 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:755 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.7126388229999998, 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0311918470000001, 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 755 2024-09-15T02:49:28.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460807+0000", 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460809+0000", 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460815+0000", 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1249999999999998e-06 2024-09-15T02:49:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460819+0000", 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9670000000000003e-06 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471918+0000", 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098856000000001 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471919+0000", 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488503+0000", 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0165840150000001 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488618+0000", 2024-09-15T02:49:28.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011474199999999999 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491998+0000", 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033803650000000002 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:766 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.7126217940000004, 2024-09-15T02:49:28.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0356599440000001, 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 766 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:28.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460824+0000", 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.27e-07 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460831+0000", 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460837+0000", 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460841+0000", 2024-09-15T02:49:28.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5889999999999999e-06 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471939+0000", 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098591 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471940+0000", 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488932+0000", 2024-09-15T02:49:28.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0169917230000001 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.489075+0000", 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014318299999999999 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.496483+0000", 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074080589999999998 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.276 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:767 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.7126007359999997, 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.037162342, 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 767 2024-09-15T02:49:28.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460845+0000", 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3200000000000001e-07 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460847+0000", 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.554e-06 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460853+0000", 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020000000000002e-06 2024-09-15T02:49:28.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460857+0000", 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0679999999999998e-06 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471961+0000", 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011103755999999999 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471962+0000", 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3600000000000003e-07 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.490940+0000", 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.018977662 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491003+0000", 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3081000000000001e-05 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.498007+0000", 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070041510000000001 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:768 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.7125838179999997, 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.038324754, 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.282 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 768 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460862+0000", 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.34e-07 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460864+0000", 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6639999999999998e-06 2024-09-15T02:49:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460870+0000", 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0859999999999998e-06 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460874+0000", 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.563e-06 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471982+0000", 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011108122999999999 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471983+0000", 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6700000000000003e-07 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492584+0000", 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0206010889999999 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492705+0000", 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012115100000000001 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.499186+0000", 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006481077 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.287 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.288 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:28.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:28.290 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:28.288+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:28.390 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:28.383+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:28.394 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:49:28.394 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":83,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:49:27.500772+0000","last_up_change":"2024-09-15T02:49:19.431893+0000","last_in_change":"2024-09-15T02:49:20.434232+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-15T02:48:36.521842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"71","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":83,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":0,"in":0,"weight":0,"primary_affinity":0.527313232421875,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":47,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":22076},{"type":"v1","addr":"172.21.15.152:6801","nonce":22076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":22076},{"type":"v1","addr":"172.21.15.152:6803","nonce":22076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":22076},{"type":"v1","addr":"172.21.15.152:6807","nonce":22076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":22076},{"type":"v1","addr":"172.21.15.152:6805","nonce":22076}]},"public_addr":"172.21.15.152:6801/22076","cluster_addr":"172.21.15.152:6803/22076","heartbeat_back_addr":"172.21.15.152:6807/22076","heartbeat_front_addr":"172.21.15.152:6805/22076","state":["exists"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":0.2740936279296875,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":29,"up_from":42,"up_thru":76,"down_at":31,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":27185},{"type":"v1","addr":"172.21.15.152:6812","nonce":27185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":27185},{"type":"v1","addr":"172.21.15.152:6819","nonce":27185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":27185},{"type":"v1","addr":"172.21.15.152:6831","nonce":27185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":27185},{"type":"v1","addr":"172.21.15.152:6826","nonce":27185}]},"public_addr":"172.21.15.152:6812/27185","cluster_addr":"172.21.15.152:6819/27185","heartbeat_back_addr":"172.21.15.152:6831/27185","heartbeat_front_addr":"172.21.15.152:6826/27185","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.496434+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.374590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.432280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.446858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.340414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.173357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-15T02:48:20.315226+0000","dead_epoch":75},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.292854+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.291714+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.339470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.593108+0000","dead_epoch":31},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.537787+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.297229+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.558699+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.578841+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.554724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":17,"length":1},{"begin":28,"length":1},{"begin":39,"length":1},{"begin":41,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:49:28.398 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:28.398 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:28.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:28.409 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T02:48:23.027168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T02:48:25.515637+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 29, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T02:48:36.521842+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '71', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 83, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}] 2024-09-15T02:49:28.409 DEBUG:teuthology.orchestra.run.smithi152:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op apply-layout-settings --pool .mgr 2024-09-15T02:49:28.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:28.491 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:28.484+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:28.513 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:28.513 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:28.513 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:28.513 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:28.513 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:28.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:328: done (5 left) 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:330: done (4 left) 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:331: done (3 left) 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:332: done (2 left) 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:333: done (1 left) 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:334: done (0 left) 2024-09-15T02:49:28.520 INFO:tasks.rados.rados.0.smithi191.stdout:335: snap_remove snap 5 2024-09-15T02:49:28.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:28.592 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:28.584+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:28.625 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/33 ECSubWriteReply(tid=19, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 46.08199132, 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74030938599999996, 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539611+0000", 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3680000000000001e-06 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539619+0000", 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-09-15T02:49:28.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278674+0000", 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73905563200000002 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278687+0000", 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.239e-05 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278784+0000", 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7149000000000001e-05 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.279919+0000", 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001135645 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=24, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:28.629 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.034624031, 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30338358999999998, 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:28.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586978+0000", 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.51e-07 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586981+0000", 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890125+0000", 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30314395700000002 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890138+0000", 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2627e-05 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890297+0000", 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015958899999999999 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890361+0000", 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3658999999999995e-05 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:285 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.023330166999997, 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019260777999999999, 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.633 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 285 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598271+0000", 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598273+0000", 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1670000000000002e-06 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598285+0000", 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1873e-05 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598308+0000", 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2895999999999999e-05 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598321+0000", 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3293e-05 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598586+0000", 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026496499999999998 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598606+0000", 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9760000000000001e-05 2024-09-15T02:49:28.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613160+0000", 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014553452 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613251+0000", 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1471000000000006e-05 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617532+0000", 2024-09-15T02:49:28.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042809010000000002 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:286 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.023167139000002, 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019362717000000002, 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.639 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 286 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598434+0000", 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598437+0000", 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4940000000000002e-06 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598451+0000", 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.448e-05 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598503+0000", 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1983e-05 2024-09-15T02:49:28.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598527+0000", 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4026e-05 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598660+0000", 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013317900000000001 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:28.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.598681+0000", 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0296e-05 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613559+0000", 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014878416 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.613646+0000", 2024-09-15T02:49:28.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6667000000000002e-05 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:48.617797+0000", 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041511760000000003 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.644 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:414 3.3s0 3:cebed992:::smithi19125844-27:head [copy-from ver 4] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.365737596999999, 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034868557000000001, 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 414 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255866+0000", 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.621e-06 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255879+0000", 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2738999999999999e-05 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255912+0000", 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3099000000000003e-05 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255927+0000", 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5013e-05 2024-09-15T02:49:28.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256006+0000", 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8688000000000003e-05 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256016+0000", 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8379999999999997e-06 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262150+0000", 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061347040000000004 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262292+0000", 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141595 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.275991+0000", 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013699266999999999 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282717+0000", 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067257910000000001 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290669+0000", 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079519480000000003 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290732+0000", 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3254000000000001e-05 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.650 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:415 3.3s0 3:cebed992:::smithi19125844-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.365649978, 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026655980999999999, 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 415 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.651 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255951+0000", 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255952+0000", 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.108e-06 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255956+0000", 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.597e-06 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255965+0000", 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3789999999999995e-06 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255971+0000", 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3199999999999999e-06 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256090+0000", 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011949 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256109+0000", 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8644999999999999e-05 2024-09-15T02:49:28.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262493+0000", 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063841519999999997 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262538+0000", 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4790999999999999e-05 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282336+0000", 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019798651 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282486+0000", 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149037 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282607+0000", 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121811 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.656 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=77, last_complete=68'67, committed=1, applied=1))", 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.209240556000001, 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.106159008, 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412362+0000", 2024-09-15T02:49:28.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9999999999999996e-07 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412370+0000", 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4120000000000001e-06 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517947+0000", 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10557704800000001 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517957+0000", 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0173999999999999e-05 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518475+0000", 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051777099999999999 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518520+0000", 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4703000000000002e-05 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:636 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~606208] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.124585409000002, 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026984608, 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 636 2024-09-15T02:49:28.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497019+0000", 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.066e-06 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497546+0000", 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052673800000000001 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497580+0000", 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4647999999999998e-05 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497592+0000", 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1669e-05 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497695+0000", 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103031 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497915+0000", 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219607 2024-09-15T02:49:28.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.499984+0000", 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020692800000000002 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518517+0000", 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018533353999999998 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523868+0000", 2024-09-15T02:49:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053511779999999998 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524001+0000", 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132037 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:637 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~688128] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.116528187, 2024-09-15T02:49:28.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022414014999999999, 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 637 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505073+0000", 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505076+0000", 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0570000000000001e-06 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505732+0000", 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065579899999999999 2024-09-15T02:49:28.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505759+0000", 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6744999999999999e-05 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.505786+0000", 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7518999999999999e-05 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.506354+0000", 2024-09-15T02:49:28.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00056810900000000002 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.506548+0000", 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000193263 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508427+0000", 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018793289999999999 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523607+0000", 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015180023000000001 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527407+0000", 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038003220000000001 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527487+0000", 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9849000000000001e-05 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:638 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~475136] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.11348581, 2024-09-15T02:49:28.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.170696291, 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 638 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508118+0000", 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6110000000000001e-06 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508908+0000", 2024-09-15T02:49:28.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00078945100000000004 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508947+0000", 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9298999999999998e-05 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508971+0000", 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4329e-05 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518663+0000", 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0096915070000000002 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518760+0000", 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7740999999999998e-05 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.519373+0000", 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061209299999999997 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523608+0000", 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042356319999999996 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678746+0000", 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15513796099999999 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678812+0000", 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5667000000000007e-05 2024-09-15T02:49:28.677 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:639 3.3s0 3:c4165a1c:::smithi19125844-22:head [setxattr _header (57) in=7b] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.112601287, 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.189219685, 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.678 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 639 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509002+0000", 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7579999999999999e-06 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509010+0000", 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3110000000000006e-06 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509031+0000", 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1234e-05 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509052+0000", 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0962e-05 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522387+0000", 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013334693 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522514+0000", 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126723 2024-09-15T02:49:28.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523057+0000", 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054339999999999998 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524433+0000", 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001375739 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698072+0000", 2024-09-15T02:49:28.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17363879099999999 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698220+0000", 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014807399999999999 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:640 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.112514055999998, 2024-09-15T02:49:28.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19189399200000001, 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 640 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509090+0000", 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2800000000000002e-06 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509095+0000", 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9679999999999997e-06 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509113+0000", 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8031000000000001e-05 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509122+0000", 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6509999999999996e-06 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524120+0000", 2024-09-15T02:49:28.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014997610999999999 2024-09-15T02:49:28.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:28.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524399+0000", 2024-09-15T02:49:28.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027887000000000002 2024-09-15T02:49:28.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698989+0000", 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.174589943 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.699138+0000", 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149333 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.700981+0000", 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018433049999999999 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=80, last_complete=68'71, committed=1, applied=1))", 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.094387527999999, 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15161024100000001, 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527215+0000", 2024-09-15T02:49:28.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.088e-06 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527221+0000", 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9909999999999999e-06 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678531+0000", 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15131061000000001 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678545+0000", 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3597e-05 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678666+0000", 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012109599999999999 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678824+0000", 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015785899999999999 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.693 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=81, last_complete=68'72, committed=1, applied=1))", 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.943125071000001, 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020448245, 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:28.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678476+0000", 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678478+0000", 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6729999999999999e-06 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678489+0000", 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.059e-05 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697765+0000", 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0192768 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697777+0000", 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1749000000000001e-05 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.697916+0000", 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013848199999999999 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698925+0000", 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010089509999999999 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:699 3.12s0 3:4a71560d:::smithi19125844-16:head [rollback 24] snapc 26=[26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.152862717, 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.064459370000000002, 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 699 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468741+0000", 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0420000000000001e-06 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468749+0000", 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000002e-06 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468764+0000", 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5353999999999999e-05 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468773+0000", 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7649999999999999e-06 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495474+0000", 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026701740000000002 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495855+0000", 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038018500000000001 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512214+0000", 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016359598 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512474+0000", 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025961499999999999 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.533198+0000", 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020724131 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:703 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.152820905, 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056538106999999997, 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 703 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468781+0000", 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468786+0000", 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4909999999999999e-06 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468794+0000", 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0229999999999992e-06 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468803+0000", 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1910000000000006e-06 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.500871+0000", 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032068169000000001 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502024+0000", 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001152674 2024-09-15T02:49:28.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525089+0000", 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023065130999999999 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525319+0000", 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022968700000000001 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:704 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.152786969999999, 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056656818999999997, 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 704 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468815+0000", 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4199999999999995e-07 2024-09-15T02:49:28.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468819+0000", 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7359999999999999e-06 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468826+0000", 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0190000000000001e-06 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468829+0000", 2024-09-15T02:49:28.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5499999999999999e-06 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502105+0000", 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033275546000000003 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502234+0000", 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012945 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525355+0000", 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023120971000000001 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525471+0000", 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011580499999999999 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.712 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:705 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.152767293, 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056845816, 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 705 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468838+0000", 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0690000000000002e-06 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468843+0000", 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7219999999999996e-06 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468849+0000", 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0340000000000002e-06 2024-09-15T02:49:28.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502534+0000", 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033684990999999997 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502672+0000", 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137701 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525531+0000", 2024-09-15T02:49:28.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022859147 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525680+0000", 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014871700000000001 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(osd.12.22:3 3.12s0 3:48b768c1:::smithi19125844-48:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:28.717 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.515968+0000", 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.105633134, 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019246940000000001, 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515968+0000", 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515968+0000", 2024-09-15T02:49:28.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515970+0000", 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9350000000000001e-06 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515977+0000", 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5139999999999998e-06 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.515995+0000", 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8425000000000001e-05 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.516003+0000", 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7959999999999996e-06 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.516081+0000", 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8308999999999999e-05 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.520820+0000", 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047385509999999997 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.535021+0000", 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014201346 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.535215+0000", 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000194064 2024-09-15T02:49:28.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 83/71 ECSubWriteReply(tid=154, last_complete=83'94, committed=1, applied=1))", 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.032065988, 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54901637599999997, 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589536+0000", 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6800000000000002e-07 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589538+0000", 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.497e-06 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138412+0000", 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54887397000000004 2024-09-15T02:49:28.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138426+0000", 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3621999999999999e-05 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138495+0000", 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9417000000000003e-05 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138552+0000", 2024-09-15T02:49:28.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6601999999999998e-05 2024-09-15T02:49:28.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.726 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:28.726 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:28.726 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:28.727 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:28.728 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:28.728 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:28.728 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:28.729 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:28.724+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:28.752 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:28.784 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:28.812 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:28.828 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:28.823+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:28.840 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:28.842 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:28.843 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:28.843 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:28.843 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:28.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:28.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:28.897 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:28.925 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:28.929 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:28.923+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:28.954 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:28.954 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:28.957 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:28.963 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:28.988 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:29.016 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:29.029 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:29.024+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:29.046 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:29.062 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.062 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:29.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:29.067+0000 7f0a137fe700 0 -- 172.21.15.6:0/2755718108 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7f0a14192730 msgr2=0x7f0a14194b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:29.084 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:29.084 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:29.084 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:29.084 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:29.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:29.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:29.130 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:29.126+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:29.178 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.178 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:29.231 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:29.228+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:29.238 INFO:tasks.rados.rados.0.smithi191.stdout:335: done (0 left) 2024-09-15T02:49:29.239 INFO:tasks.rados.rados.0.smithi191.stdout:336: delete oid 18 current snap is 30 2024-09-15T02:49:29.242 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:49:29.265 INFO:tasks.rados.rados.0.smithi191.stdout:336: done (0 left) 2024-09-15T02:49:29.265 INFO:tasks.rados.rados.0.smithi191.stdout:337: rollback oid 23 current snap is 30 2024-09-15T02:49:29.265 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 23 to 26 2024-09-15T02:49:29.265 INFO:tasks.rados.rados.0.smithi191.stdout:338: snap_create 2024-09-15T02:49:29.280 INFO:tasks.rados.rados.0.smithi191.stdout:337: finishing rollback tid 0 to smithi19125844-23 2024-09-15T02:49:29.282 INFO:tasks.rados.rados.0.smithi191.stdout:337: finishing rollback tid 1 to smithi19125844-23 2024-09-15T02:49:29.282 INFO:tasks.rados.rados.0.smithi191.stdout:337: finishing rollback tid 2 to smithi19125844-23 2024-09-15T02:49:29.282 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 23 v 100 (ObjNum 118 snap 26 seq_num 118) dirty exists 2024-09-15T02:49:29.308 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.308 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:29.331 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:29.324+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:29.417 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.417 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:29.431 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:29.430+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:29.528 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.529 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.532 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:29.527+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:29.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:29.558 INFO:teuthology.orchestra.run.smithi152.stderr:Nothing to do for non-filestore backend 2024-09-15T02:49:29.633 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:29.628+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:29.688 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.688 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:29.733 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:29.732+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:29.794 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.795 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:29.834 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:29.832+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:29.906 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:29.906 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:29.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:29.936 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:29.930+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:30.018 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.019 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:30.035 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:30.027+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:30.107 DEBUG:teuthology.orchestra.run.smithi152:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op apply-layout-settings --pool rbd 2024-09-15T02:49:30.129 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.130 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.136 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:30.131+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:30.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:30.237 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:30.232+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:30.239 INFO:tasks.rados.rados.0.smithi191.stdout:337: done (1 left) 2024-09-15T02:49:30.239 INFO:tasks.rados.rados.0.smithi191.stdout:338: done (0 left) 2024-09-15T02:49:30.239 INFO:tasks.rados.rados.0.smithi191.stdout:339: copy_from oid 24 from oid 38 current snap is 31 2024-09-15T02:49:30.239 INFO:tasks.rados.rados.0.smithi191.stdout:340: snap_create 2024-09-15T02:49:30.285 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.285 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:30.338 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:30.332+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:30.389 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.389 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:30.439 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:30.432+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:30.497 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.497 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:30.552 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:30.548+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:30.614 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.614 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:30.641 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:30.638+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:30.724 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.724 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:30.742 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:30.738+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:30.832 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.832 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:30.842 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:30.835+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:30.940 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:30.940 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:30.945 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:30.942+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:30.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:31.045 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:31.045 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:31.046 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:31.042+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:31.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:31.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:31.115 INFO:tasks.rados.rados.0.smithi191.stdout:339: finishing copy_from racing read to smithi19125844-24 2024-09-15T02:49:31.116 INFO:tasks.rados.rados.0.smithi191.stdout:339: finishing copy_from to smithi19125844-24 2024-09-15T02:49:31.116 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 24 v 50 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T02:49:31.149 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:31.149 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:31.150 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:31.143+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:31.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:31.242 INFO:tasks.rados.rados.0.smithi191.stdout:339: done (1 left) 2024-09-15T02:49:31.242 INFO:tasks.rados.rados.0.smithi191.stdout:340: done (0 left) 2024-09-15T02:49:31.242 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 9 current snap is 32 2024-09-15T02:49:31.242 INFO:tasks.rados.rados.0.smithi191.stdout:341: seq_num 125 ranges {1015808=557056,1572864=475136,2048000=507904,2555904=475136,3031040=98304} 2024-09-15T02:49:31.250 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:31.244+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:31.253 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:31.253 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:31.254 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:31.258 INFO:tasks.rados.rados.0.smithi191.stdout:341: writing smithi19125844-9 from 1015808 to 1572864 tid 1 2024-09-15T02:49:31.261 INFO:tasks.rados.rados.0.smithi191.stdout:341: writing smithi19125844-9 from 1572864 to 2048000 tid 2 2024-09-15T02:49:31.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:31.263 INFO:tasks.rados.rados.0.smithi191.stdout:341: writing smithi19125844-9 from 2048000 to 2555904 tid 3 2024-09-15T02:49:31.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:49:31.267 INFO:tasks.rados.rados.0.smithi191.stdout:341: writing smithi19125844-9 from 2555904 to 3031040 tid 4 2024-09-15T02:49:31.267 INFO:tasks.rados.rados.0.smithi191.stdout:341: writing smithi19125844-9 from 3031040 to 3129344 tid 5 2024-09-15T02:49:31.267 INFO:tasks.rados.rados.0.smithi191.stdout:342: rollback oid 49 current snap is 32 2024-09-15T02:49:31.267 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 49 to 26 2024-09-15T02:49:31.267 INFO:tasks.rados.rados.0.smithi191.stdout:343: setattr oid 40 current snap is 32 2024-09-15T02:49:31.269 INFO:tasks.rados.rados.0.smithi191.stdout:344: read oid 10 snap -1 2024-09-15T02:49:31.269 INFO:tasks.rados.rados.0.smithi191.stdout:344: expect (ObjNum 114 snap 23 seq_num 114) 2024-09-15T02:49:31.269 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 25 current snap is 32 2024-09-15T02:49:31.269 INFO:tasks.rados.rados.0.smithi191.stdout:345: seq_num 126 ranges {1671168=573440,2244608=622592,2867200=622592,3489792=311296} 2024-09-15T02:49:31.282 INFO:tasks.rados.rados.0.smithi191.stdout:345: writing smithi19125844-25 from 1671168 to 2244608 tid 1 2024-09-15T02:49:31.286 INFO:tasks.rados.rados.0.smithi191.stdout:345: writing smithi19125844-25 from 2244608 to 2867200 tid 2 2024-09-15T02:49:31.290 INFO:tasks.rados.rados.0.smithi191.stdout:345: writing smithi19125844-25 from 2867200 to 3489792 tid 3 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:345: writing smithi19125844-25 from 3489792 to 3801088 tid 4 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:346: rollback oid 37 current snap is 32 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 37 to 17 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:347: read oid 46 snap -1 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:347: expect deleted 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:348: read oid 21 snap -1 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:348: expect (ObjNum 106 snap 19 seq_num 106) 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:349: copy_from oid 43 from oid 8 current snap is 32 2024-09-15T02:49:31.293 INFO:tasks.rados.rados.0.smithi191.stdout:350: copy_from oid 12 from oid 30 current snap is 32 2024-09-15T02:49:31.294 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 16 current snap is 32 2024-09-15T02:49:31.294 INFO:tasks.rados.rados.0.smithi191.stdout:351: seq_num 127 ranges {0=589824,589824=753664,1343488=475136,1818624=327680} 2024-09-15T02:49:31.296 INFO:tasks.rados.rados.0.smithi191.stdout:351: writing smithi19125844-16 from 0 to 589824 tid 1 2024-09-15T02:49:31.297 INFO:teuthology.orchestra.run.smithi152.stderr:Nothing to do for non-filestore backend 2024-09-15T02:49:31.297 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:31.301 INFO:tasks.rados.rados.0.smithi191.stdout:351: writing smithi19125844-16 from 589824 to 1343488 tid 2 2024-09-15T02:49:31.304 INFO:tasks.rados.rados.0.smithi191.stdout:351: writing smithi19125844-16 from 1343488 to 1818624 tid 3 2024-09-15T02:49:31.307 INFO:tasks.rados.rados.0.smithi191.stdout:351: writing smithi19125844-16 from 1818624 to 2146304 tid 4 2024-09-15T02:49:31.307 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 28 current snap is 32 2024-09-15T02:49:31.307 INFO:tasks.rados.rados.0.smithi191.stdout:342: finishing rollback tid 0 to smithi19125844-49 2024-09-15T02:49:31.307 INFO:tasks.rados.rados.0.smithi191.stdout:352: seq_num 128 ranges {704512=606208,1310720=507904,1818624=573440,2392064=475136} 2024-09-15T02:49:31.315 INFO:tasks.rados.rados.0.smithi191.stdout:352: writing smithi19125844-28 from 704512 to 1310720 tid 1 2024-09-15T02:49:31.319 INFO:tasks.rados.rados.0.smithi191.stdout:352: writing smithi19125844-28 from 1310720 to 1818624 tid 2 2024-09-15T02:49:31.323 INFO:tasks.rados.rados.0.smithi191.stdout:352: writing smithi19125844-28 from 1818624 to 2392064 tid 3 2024-09-15T02:49:31.326 INFO:tasks.rados.rados.0.smithi191.stdout:352: writing smithi19125844-28 from 2392064 to 2867200 tid 4 2024-09-15T02:49:31.326 INFO:tasks.rados.rados.0.smithi191.stdout:353: read oid 22 snap -1 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:353: expect (ObjNum 105 snap 18 seq_num 105) 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:354: snap_create 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:342: finishing rollback tid 1 to smithi19125844-49 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:342: finishing rollback tid 2 to smithi19125844-49 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 49 v 66 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 40 v 67 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-15T02:49:31.327 INFO:tasks.rados.rados.0.smithi191.stdout:344: expect (ObjNum 114 snap 23 seq_num 114) 2024-09-15T02:49:31.351 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:31.343+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:31.370 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:31.387 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:31.387 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:31.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:31.397 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:31.400 INFO:tasks.rados.rados.0.smithi191.stdout:348: expect (ObjNum 106 snap 19 seq_num 106) 2024-09-15T02:49:31.425 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:31.451 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:31.452 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:31.448+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 1 to smithi19125844-9 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:345: finishing write tid 1 to smithi19125844-25 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 2 to smithi19125844-9 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 3 to smithi19125844-9 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 4 to smithi19125844-9 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:345: finishing write tid 2 to smithi19125844-25 2024-09-15T02:49:31.461 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 5 to smithi19125844-9 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:351: finishing write tid 1 to smithi19125844-16 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:345: finishing write tid 3 to smithi19125844-25 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:345: finishing write tid 4 to smithi19125844-25 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 6 to smithi19125844-9 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:345: finishing write tid 5 to smithi19125844-25 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:351: finishing write tid 2 to smithi19125844-16 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:351: finishing write tid 3 to smithi19125844-16 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:345: finishing write tid 6 to smithi19125844-25 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 25 v 62 (ObjNum 126 snap 32 seq_num 126) dirty exists 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:345: left oid 25 (ObjNum 126 snap 32 seq_num 126) 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:341: finishing write tid 7 to smithi19125844-9 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 9 v 117 (ObjNum 125 snap 32 seq_num 125) dirty exists 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:341: left oid 9 (ObjNum 125 snap 32 seq_num 125) 2024-09-15T02:49:31.462 INFO:tasks.rados.rados.0.smithi191.stdout:353: expect (ObjNum 105 snap 18 seq_num 105) 2024-09-15T02:49:31.478 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:31.487 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:31.487 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:31.496 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:31.502 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:31.525 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:31.529 INFO:tasks.rados.rados.0.smithi191.stdout:350: finishing copy_from racing read to smithi19125844-12 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:346: finishing rollback tid 0 to smithi19125844-37 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:346: finishing rollback tid 1 to smithi19125844-37 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:346: finishing rollback tid 2 to smithi19125844-37 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 37 v 109 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:351: finishing write tid 4 to smithi19125844-16 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:352: finishing write tid 1 to smithi19125844-28 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:352: finishing write tid 2 to smithi19125844-28 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:352: finishing write tid 3 to smithi19125844-28 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:352: finishing write tid 4 to smithi19125844-28 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:352: finishing write tid 5 to smithi19125844-28 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:351: finishing write tid 5 to smithi19125844-16 2024-09-15T02:49:31.530 INFO:tasks.rados.rados.0.smithi191.stdout:352: finishing write tid 6 to smithi19125844-28 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 72 (ObjNum 128 snap 32 seq_num 128) dirty exists 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:352: left oid 28 (ObjNum 128 snap 32 seq_num 128) 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:351: finishing write tid 6 to smithi19125844-16 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 16 v 54 (ObjNum 127 snap 32 seq_num 127) dirty exists 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:351: left oid 16 (ObjNum 127 snap 32 seq_num 127) 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:350: finishing copy_from to smithi19125844-12 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 12 v 90 (ObjNum 117 snap 23 seq_num 117) dirty exists 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:349: finishing copy_from racing read to smithi19125844-43 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:349: finishing copy_from to smithi19125844-43 2024-09-15T02:49:31.531 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 108 (ObjNum 121 snap 29 seq_num 121) dirty exists 2024-09-15T02:49:31.553 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:31.548+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:31.557 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:31.585 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:31.593 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:31.593 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:31.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:31.613 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:31.637 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:31.654 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:31.652+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:31.666 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:31.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:31.667+0000 7fbb9a7fc700 0 -- 172.21.15.6:0/529853589 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fbb9c1923f0 msgr2=0x7fbb9c1947d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:31.696 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:31.696 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:31.697 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:31.697 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:31.697 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:31.702 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:31.702 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:31.702 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:31.702 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:31.702 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:31.702 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:31.704 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:31.705 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:31.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:31.755 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:31.748+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:388 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:31.803 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.048050953999997, 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.205927366, 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 388 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752805+0000", 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8480000000000003e-06 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752820+0000", 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752827+0000", 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8059999999999997e-06 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770449+0000", 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017621527000000001 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770567+0000", 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118576 2024-09-15T02:49:31.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776872+0000", 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063044340000000003 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776986+0000", 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011429699999999999 2024-09-15T02:49:31.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958728+0000", 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.181741654 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:389 3.7s0 3:e10d9d4d:::smithi19125844-8:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.048015693000004, 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47945425899999999, 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 389 2024-09-15T02:49:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752836+0000", 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752842+0000", 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8579999999999996e-06 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752860+0000", 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7816e-05 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752867+0000", 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0260000000000002e-06 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770793+0000", 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017925963999999999 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770931+0000", 2024-09-15T02:49:31.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137978 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955584+0000", 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.184652765 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.959571+0000", 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003987488 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:31.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232070+0000", 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.272499243 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232290+0000", 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021937500000000001 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:390 3.7s0 3:e10d9d4d:::smithi19125844-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:31.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.047977502999998, 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20558775300000001, 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 390 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8199999999999999e-07 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752878+0000", 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4489999999999999e-06 2024-09-15T02:49:31.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752888+0000", 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0122000000000001e-05 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752894+0000", 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9220000000000002e-06 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771143+0000", 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018248665000000001 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771198+0000", 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5158000000000003e-05 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958258+0000", 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18706018699999999 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958359+0000", 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101025 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958461+0000", 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102543 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:402 3.1s0 3:801a1da7:::smithi19125844-39:e [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:31.820 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.04259974, 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52960550699999998, 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 402 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:31.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758252+0000", 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5000000000000002e-07 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758257+0000", 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3800000000000004e-06 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758267+0000", 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0455000000000001e-05 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758604+0000", 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033705300000000003 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761065+0000", 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024613840000000001 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761208+0000", 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014282499999999999 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287857+0000", 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52664865999999999 2024-09-15T02:49:31.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 49/33 ECSubReadReply(tid=183, attrs_read=0))", 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.037627239999999, 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52471420199999996, 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763572+0000", 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034791800000000001 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286378+0000", 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52280558799999999 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286384+0000", 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286558+0000", 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017370499999999999 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287938+0000", 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013805550000000001 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:433 3.7s0 3:ec23d144:::smithi19125844-41:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.517857867000004, 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27191602199999998, 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.829 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282994+0000", 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1790000000000001e-06 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283000+0000", 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0479999999999998e-06 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283008+0000", 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1440000000000003e-06 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283012+0000", 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7129999999999996e-06 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298201+0000", 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015188495 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298355+0000", 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015441200000000001 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298951+0000", 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059618300000000002 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300677+0000", 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001725285 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554804+0000", 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25412710900000002 2024-09-15T02:49:31.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554909+0000", 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105454 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:434 3.7s0 3:ec23d144:::smithi19125844-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.517833944000003, 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27651220700000001, 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.834 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 434 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283018+0000", 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0500000000000003e-07 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283021+0000", 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.959e-06 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283028+0000", 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0169999999999999e-06 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283032+0000", 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0509999999999998e-06 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299373+0000", 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016340659 2024-09-15T02:49:31.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299481+0000", 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108218 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554991+0000", 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25551037700000001 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.555256+0000", 2024-09-15T02:49:31.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026495400000000002 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559529+0000", 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042733670000000001 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.839 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:437 3.6s0 3:6e9f7d10:::smithi19125844-33:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.517767007000003, 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20344801100000001, 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 437 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283085+0000", 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283088+0000", 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.993e-06 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283094+0000", 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5810000000000002e-06 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6289999999999999e-06 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284035+0000", 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093763400000000002 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284100+0000", 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4882999999999995e-05 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486532+0000", 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20243185599999999 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.843 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:438 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [delete] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e50)", 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.517733198000002, 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27354936000000002, 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 438 2024-09-15T02:49:31.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8100000000000003e-07 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283121+0000", 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8830000000000002e-06 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283129+0000", 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3390000000000004e-06 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283133+0000", 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299523+0000", 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389989000000001 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299625+0000", 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010208 2024-09-15T02:49:31.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300116+0000", 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049149000000000003 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300972+0000", 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085554200000000004 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556582+0000", 2024-09-15T02:49:31.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25561025399999998 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556667+0000", 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5208000000000001e-05 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.6s0 50/33 ECSubReadReply(tid=200, attrs_read=0))", 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.505563604999999, 2024-09-15T02:49:31.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19127902199999999, 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2600000000000001e-07 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295338+0000", 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9951999999999999e-05 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485520+0000", 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19018270600000001 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485530+0000", 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0820000000000005e-06 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485655+0000", 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012580999999999999 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486567+0000", 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000911246 2024-09-15T02:49:31.852 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=203, last_complete=50'62, committed=1, applied=1))", 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.497518423000002, 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25159346799999999, 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303336+0000", 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9550000000000001e-06 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554637+0000", 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25130088299999997 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554647+0000", 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9329999999999996e-06 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554704+0000", 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6514000000000001e-05 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554926+0000", 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222647 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.856 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 50/33 ECSubWriteReply(tid=205, last_complete=50'35, committed=1, applied=1))", 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.481510993000001, 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.23186345, 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319341+0000", 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6700000000000003e-07 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319347+0000", 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6879999999999996e-06 2024-09-15T02:49:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550925+0000", 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.231578591 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550946+0000", 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0903e-05 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551110+0000", 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016408300000000001 2024-09-15T02:49:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551204+0000", 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3418000000000006e-05 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=208, last_complete=50'64, committed=1, applied=1))", 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.241653827, 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.60425413100000003, 2024-09-15T02:49:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559198+0000", 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9500000000000001e-07 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559203+0000", 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.13e-06 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163222+0000", 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.604018638 2024-09-15T02:49:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163237+0000", 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4622e-05 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163384+0000", 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014714099999999999 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163451+0000", 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7805000000000006e-05 2024-09-15T02:49:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 54/53 ECSubWrite(tid=87, reqid=client.4526.0:461, at_version=0'0, trim_to=0'0, roll_forward_to=54'39))", 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.879751159999998, 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.41018363000000002, 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921102+0000", 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921127+0000", 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4987e-05 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329820+0000", 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.40869367400000001 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329833+0000", 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2669e-05 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329958+0000", 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012490099999999999 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329981+0000", 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2463000000000002e-05 2024-09-15T02:49:31.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331170+0000", 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001189701 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331284+0000", 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113435 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.868 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPushReply(3.1s0 54/53 [PushReplyOp(3:801a1da7:::smithi19125844-39:head)])", 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.506034376999999, 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99244340499999995, 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294818+0000", 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294837+0000", 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9052999999999999e-05 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286615+0000", 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99177708899999995 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286629+0000", 2024-09-15T02:49:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4583000000000001e-05 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286721+0000", 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1640999999999999e-05 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.287260+0000", 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053939199999999997 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 55/53 ECSubRead(tid=88, to_read={3:94f52c52:::smithi19125844-20:head=0,110592,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.470963904000001, 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59246917700000001, 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329888+0000", 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329900+0000", 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.188e-05 2024-09-15T02:49:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921816+0000", 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59191607099999999 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921831+0000", 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4765e-05 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922011+0000", 2024-09-15T02:49:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018023900000000001 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922356+0000", 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034564200000000001 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 62/53 ECSubWriteReply(tid=316, last_complete=62'23, committed=1, applied=1))", 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.443409407000001, 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58891104599999999, 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6400000000000002e-07 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357447+0000", 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9529999999999998e-06 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945262+0000", 2024-09-15T02:49:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58781430300000004 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945281+0000", 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9366000000000001e-05 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945429+0000", 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014825600000000001 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.946353+0000", 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092350400000000001 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.0.22:2 3.7s0 3:e10d9d4d:::smithi19125844-8:head [assert-version v25,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.324933717, 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.014934445, 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475920+0000", 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475928+0000", 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1750000000000005e-06 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475952+0000", 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4244e-05 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475960+0000", 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0940000000000008e-06 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476051+0000", 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1032000000000005e-05 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476057+0000", 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3029999999999999e-06 2024-09-15T02:49:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484706+0000", 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0086496460000001 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484954+0000", 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024785999999999997 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.490852+0000", 2024-09-15T02:49:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058976979999999998 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:826 3.7s0 3:e10d9d4d:::smithi19125844-8:head [setxattr _header (56) in=7b] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.548153+0000", 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.2526979789999997, 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.121709972, 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.885 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 826 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548153+0000", 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548153+0000", 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548154+0000", 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0200000000000001e-07 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548158+0000", 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7019999999999999e-06 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548164+0000", 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2369999999999997e-06 2024-09-15T02:49:31.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548168+0000", 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1339999999999997e-06 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.567926+0000", 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019757756000000001 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.568159+0000", 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023356900000000001 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.569076+0000", 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091681900000000003 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570151+0000", 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001075028 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.669693+0000", 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.099542017999999996 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.669863+0000", 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017000699999999999 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:827 3.7s0 3:e10d9d4d:::smithi19125844-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.548172+0000", 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.2526791040000003, 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.126680192, 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 827 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548172+0000", 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548172+0000", 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548173+0000", 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5799999999999999e-07 2024-09-15T02:49:31.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548175+0000", 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3539999999999998e-06 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548181+0000", 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9100000000000002e-06 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:31.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548185+0000", 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8139999999999998e-06 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570054+0000", 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021869748000000001 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570139+0000", 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4345000000000006e-05 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.670845+0000", 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.100706474 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.671033+0000", 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018782299999999999 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.674852+0000", 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038191660000000001 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.895 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.896 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.896 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:31.896 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:31.896 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:31.898 DEBUG:teuthology.orchestra.run.smithi152:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op apply-layout-settings --pool unique_pool_0 2024-09-15T02:49:31.898 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:31.899 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:31.892+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:31.998 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:31.996+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:32.060 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:32.060 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:32.060 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:32.060 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:32.069 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:32.099 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:32.092+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:32.162 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:32.162 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:32.162 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:32.162 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:32.162 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:32.173 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:32.200 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:32.192+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:341: done (13 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:342: done (12 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:343: done (11 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:344: done (10 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:345: done (9 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:346: done (8 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:347: done (7 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:348: done (6 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:349: done (5 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:350: done (4 left) 2024-09-15T02:49:32.244 INFO:tasks.rados.rados.0.smithi191.stdout:351: done (3 left) 2024-09-15T02:49:32.245 INFO:tasks.rados.rados.0.smithi191.stdout:352: done (2 left) 2024-09-15T02:49:32.245 INFO:tasks.rados.rados.0.smithi191.stdout:353: done (1 left) 2024-09-15T02:49:32.245 INFO:tasks.rados.rados.0.smithi191.stdout:354: done (0 left) 2024-09-15T02:49:32.245 INFO:tasks.rados.rados.0.smithi191.stdout:355: delete oid 29 current snap is 33 2024-09-15T02:49:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:49:32.256 INFO:tasks.rados.rados.0.smithi191.stdout:355: done (0 left) 2024-09-15T02:49:32.256 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 36 current snap is 33 2024-09-15T02:49:32.256 INFO:tasks.rados.rados.0.smithi191.stdout:356: seq_num 130 ranges {2408448=638976,3047424=671744,3719168=802816,4521984=81920} 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:49:32.272 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 67.533178200999998, 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.273 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.274 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:49:32.275 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.276 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:403 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~606208] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.512439507000003, 2024-09-15T02:49:32.277 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.195887642, 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 403 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.278 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757022+0000", 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.593e-06 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757800+0000", 2024-09-15T02:49:32.279 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077824999999999997 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757830+0000", 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0057e-05 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757844+0000", 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3947999999999999e-05 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.280 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757925+0000", 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0249999999999999e-05 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.758082+0000", 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015777599999999999 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759386+0000", 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001303217 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777348+0000", 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017962568000000002 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952681+0000", 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17533264000000001 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952907+0000", 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000226343 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.282 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:404 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~720896] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.508220903000002, 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.192160255, 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 404 2024-09-15T02:49:32.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761280+0000", 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2138999999999998e-05 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762400+0000", 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001119451 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762433+0000", 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3238000000000003e-05 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762446+0000", 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3308e-05 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768118+0000", 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056721360000000004 2024-09-15T02:49:32.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768285+0000", 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000166482 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769662+0000", 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013768140000000001 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780475+0000", 2024-09-15T02:49:32.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010813578000000001 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953275+0000", 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17280025199999999 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953398+0000", 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012285700000000001 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.288 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:405 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~49152] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.506990164999998, 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19129260000000001, 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 405 2024-09-15T02:49:32.289 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762470+0000", 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4699999999999999e-06 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762527+0000", 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7151999999999998e-05 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762550+0000", 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2311999999999999e-05 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762559+0000", 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777614+0000", 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015055628999999999 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777696+0000", 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1840000000000002e-05 2024-09-15T02:49:32.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778120+0000", 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042372900000000002 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780527+0000", 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00240712 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953622+0000", 2024-09-15T02:49:32.293 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17309514500000001 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953761+0000", 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013926300000000001 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:406 3.es0 3:71c1a91c:::smithi19125844-45:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:32.294 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.506890204999998, 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19225968700000001, 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 406 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762570+0000", 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.034e-06 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762575+0000", 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4079999999999997e-06 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762592+0000", 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.715e-05 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762600+0000", 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6480000000000003e-06 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779660+0000", 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017060035000000001 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779757+0000", 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6537000000000001e-05 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780137+0000", 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038070299999999998 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781414+0000", 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012764230000000001 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954649+0000", 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17323481900000001 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954828+0000", 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017992999999999999 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.299 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:407 3.es0 3:71c1a91c:::smithi19125844-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.506850333000003, 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19532122900000001, 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 407 2024-09-15T02:49:32.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.301 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762610+0000", 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762614+0000", 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0659999999999997e-06 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762626+0000", 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2490000000000001e-05 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762639+0000", 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2704000000000001e-05 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781806+0000", 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019166823999999999 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.782082+0000", 2024-09-15T02:49:32.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027635000000000002 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955680+0000", 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17359843999999999 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955807+0000", 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012658299999999999 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.304 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.957930+0000", 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021228499999999999 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.13.22:5 3.es0 3:740f3d5b:::smithi19125844-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:32.305 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.494319353000002, 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.177222558, 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.306 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775141+0000", 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775146+0000", 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5630000000000002e-06 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.307 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775159+0000", 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3509999999999999e-05 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775165+0000", 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8130000000000001e-06 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.308 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778738+0000", 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003572867 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778948+0000", 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00021028299999999999 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952362+0000", 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17341393199999999 2024-09-15T02:49:32.309 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:529 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~802816] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.014633895999999, 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.030230086, 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.310 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 529 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:32.311 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254827+0000", 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6390000000000001e-06 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256167+0000", 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013402900000000001 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.312 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256196+0000", 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8762999999999999e-05 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256203+0000", 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6229999999999997e-06 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036663860000000002 2024-09-15T02:49:32.313 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.260012+0000", 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014185099999999999 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017728349999999999 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277478+0000", 2024-09-15T02:49:32.314 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015693650999999999 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285001+0000", 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075225120000000003 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.315 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285055+0000", 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4536000000000001e-05 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:530 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~638976] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.010712741999999, 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026488489, 2024-09-15T02:49:32.316 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 530 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.317 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258748+0000", 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.66e-06 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259495+0000", 2024-09-15T02:49:32.318 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00074681400000000005 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259522+0000", 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7552999999999999e-05 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259538+0000", 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6220000000000001e-05 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.319 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269841+0000", 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010302477000000001 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269927+0000", 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5993000000000002e-05 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.270621+0000", 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00069455099999999996 2024-09-15T02:49:32.320 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277769+0000", 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0071479769999999998 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285179+0000", 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074099320000000002 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.321 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285235+0000", 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5312000000000002e-05 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:533 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.009754048000001, 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027459596999999999, 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.322 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 533 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.323 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.89e-07 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259707+0000", 2024-09-15T02:49:32.324 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5460000000000001e-06 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259711+0000", 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1439999999999996e-06 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259715+0000", 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3340000000000003e-06 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.325 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277889+0000", 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018173363000000001 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277948+0000", 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8913999999999999e-05 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286109+0000", 2024-09-15T02:49:32.326 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0081616199999999996 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286156+0000", 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6928000000000002e-05 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.287165+0000", 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010083589999999999 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.327 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s2 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.735357918, 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027643175999999998, 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:32.328 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534111+0000", 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8090000000000004e-06 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.329 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534425+0000", 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031419599999999998 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542230+0000", 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0078045069999999996 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542239+0000", 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.330 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542375+0000", 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000135897 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542396+0000", 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1237999999999999e-05 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561688+0000", 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019291689000000001 2024-09-15T02:49:32.331 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561744+0000", 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6178999999999999e-05 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.8s2 77/76 ECSubRead(tid=258, to_read={3:173bfaf3:::smithi19125844-44:18=262144,262144,0}, subchunks={3:173bfaf3:::smithi19125844-44:18=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.69359916, 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119859681, 2024-09-15T02:49:32.332 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.333 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575861+0000", 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0640000000000001e-06 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575876+0000", 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5165e-05 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694960+0000", 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119084367 2024-09-15T02:49:32.334 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694980+0000", 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9930000000000001e-05 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695069+0000", 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8737000000000001e-05 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695719+0000", 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065041800000000005 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=112, attrs_read=0))", 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.689109127, 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13363014300000001, 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580351+0000", 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2419999999999999e-06 2024-09-15T02:49:32.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580721+0000", 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00037001299999999998 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712609+0000", 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13188792199999999 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712623+0000", 2024-09-15T02:49:32.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.429e-05 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712782+0000", 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000158791 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.713980+0000", 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011978850000000001 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.522081882, 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21641586800000001, 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.340 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747379+0000", 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747806+0000", 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042687399999999999 2024-09-15T02:49:32.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962958+0000", 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21515266799999999 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962974+0000", 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6237999999999999e-05 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963059+0000", 2024-09-15T02:49:32.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4968000000000004e-05 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963793+0000", 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00073351999999999998 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.12s5 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 9.7476168370000007, 2024-09-15T02:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028389666000000001, 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.344 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521843+0000", 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.122e-06 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522442+0000", 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00059853399999999996 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533671+0000", 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011229075 2024-09-15T02:49:32.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533686+0000", 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5265999999999999e-05 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533798+0000", 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000112198 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.346 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533821+0000", 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2229999999999999e-05 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550165+0000", 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016344826 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550232+0000", 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6414999999999993e-05 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.347 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:836 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~573440] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.6708225409999997, 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033215366000000003, 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 836 2024-09-15T02:49:32.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598639+0000", 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1280000000000002e-06 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599731+0000", 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001092469 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599759+0000", 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8472000000000001e-05 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.350 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599774+0000", 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4612e-05 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608764+0000", 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089903399999999994 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608966+0000", 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020188699999999999 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.610666+0000", 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016996940000000001 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.630970+0000", 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020303799000000001 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631771+0000", 2024-09-15T02:49:32.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00080119100000000003 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631852+0000", 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0773999999999996e-05 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.353 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:837 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~688128] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.6659070620000005, 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037719309999999999, 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 837 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603553+0000", 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.355 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604429+0000", 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000875577 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604454+0000", 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4794999999999999e-05 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604464+0000", 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9199999999999999e-06 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.356 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621603+0000", 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017138823000000001 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621666+0000", 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2996000000000005e-05 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.622382+0000", 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071611400000000001 2024-09-15T02:49:32.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631868+0000", 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0094864259999999992 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641117+0000", 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0092490960000000001 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641271+0000", 2024-09-15T02:49:32.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015396299999999999 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:838 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~294912] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.6639648740000004, 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.03629342, 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.359 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 838 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605496+0000", 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.415e-06 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605778+0000", 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028258100000000001 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605797+0000", 2024-09-15T02:49:32.361 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8817999999999998e-05 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605805+0000", 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5229999999999995e-06 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626744+0000", 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020938365 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.362 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626872+0000", 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012857999999999999 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.627645+0000", 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077218100000000004 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631967+0000", 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043226499999999999 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641643+0000", 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096753510000000004 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641788+0000", 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014495600000000001 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.364 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:839 3.es0 3:7773fe95:::smithi19125844-37:head [setxattr _header (57) in=7b] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.6636447939999996, 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.036686074999999999, 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 839 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.365 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605815+0000", 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1909999999999999e-06 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605822+0000", 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3369999999999996e-06 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605840+0000", 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7949e-05 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605848+0000", 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6980000000000004e-06 2024-09-15T02:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631010+0000", 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025161701000000002 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631081+0000", 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0993000000000003e-05 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631379+0000", 2024-09-15T02:49:32.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000298146 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.632106+0000", 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072663800000000005 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642331+0000", 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010225316 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642500+0000", 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000169106 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:840 3.es0 3:7773fe95:::smithi19125844-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 4.6636011709999998, 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.039020892000000001, 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.370 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 840 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:32.371 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605859+0000", 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.02e-06 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605864+0000", 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1000000000000003e-06 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605882+0000", 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8128999999999999e-05 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605891+0000", 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5550000000000003e-06 2024-09-15T02:49:32.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631895+0000", 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026004328 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.376 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631958+0000", 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2603999999999999e-05 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643304+0000", 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01134697 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643450+0000", 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000145069 2024-09-15T02:49:32.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.644879+0000", 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0014291169999999999 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.378 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:32.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:356: writing smithi19125844-36 from 2408448 to 3047424 tid 1 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:356: writing smithi19125844-36 from 3047424 to 3719168 tid 2 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:356: writing smithi19125844-36 from 3719168 to 4521984 tid 3 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:356: writing smithi19125844-36 from 4521984 to 4603904 tid 4 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 21 current snap is 33 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:357: seq_num 131 ranges {2129920=655360,2785280=425984,3211264=638976,3850240=491520} 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:357: writing smithi19125844-21 from 2129920 to 2785280 tid 1 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:357: writing smithi19125844-21 from 2785280 to 3211264 tid 2 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:357: writing smithi19125844-21 from 3211264 to 3850240 tid 3 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:357: writing smithi19125844-21 from 3850240 to 4341760 tid 4 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:358: snap_create 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:356: finishing write tid 1 to smithi19125844-36 2024-09-15T02:49:32.383 INFO:tasks.rados.rados.0.smithi191.stdout:356: finishing write tid 2 to smithi19125844-36 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:356: finishing write tid 3 to smithi19125844-36 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:356: finishing write tid 4 to smithi19125844-36 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:356: finishing write tid 5 to smithi19125844-36 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:356: finishing write tid 6 to smithi19125844-36 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 36 v 73 (ObjNum 130 snap 33 seq_num 130) dirty exists 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:356: left oid 36 (ObjNum 130 snap 33 seq_num 130) 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:357: finishing write tid 1 to smithi19125844-21 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:357: finishing write tid 2 to smithi19125844-21 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:357: finishing write tid 3 to smithi19125844-21 2024-09-15T02:49:32.384 INFO:tasks.rados.rados.0.smithi191.stdout:357: finishing write tid 4 to smithi19125844-21 2024-09-15T02:49:32.385 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:32.372+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:32.481 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:32.475+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:32.486 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:32.486 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:32.486 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:32.486 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:32.495 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:32.498 INFO:tasks.rados.rados.0.smithi191.stdout:357: finishing write tid 5 to smithi19125844-21 2024-09-15T02:49:32.501 INFO:tasks.rados.rados.0.smithi191.stdout:357: finishing write tid 6 to smithi19125844-21 2024-09-15T02:49:32.502 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 21 v 113 (ObjNum 131 snap 33 seq_num 131) dirty exists 2024-09-15T02:49:32.502 INFO:tasks.rados.rados.0.smithi191.stdout:357: left oid 21 (ObjNum 131 snap 33 seq_num 131) 2024-09-15T02:49:32.581 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:32.576+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:32.593 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:32.594 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:32.594 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:32.594 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:32.594 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:32.603 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:32.683 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:32.680+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.079495565999999, 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:49:32.701 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.702 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:49:32.703 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.704 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:416 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~638976] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.437287077000001, 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026381341999999999, 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.705 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 416 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.706 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261066+0000", 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1159999999999999e-06 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262194+0000", 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001127783 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.707 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262245+0000", 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0880000000000001e-05 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262260+0000", 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5065999999999999e-05 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262382+0000", 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012151599999999999 2024-09-15T02:49:32.708 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262403+0000", 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1344000000000001e-05 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266706+0000", 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004303355 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.709 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266794+0000", 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7717000000000001e-05 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268260+0000", 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001466406 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285160+0000", 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016899773 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.710 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287372+0000", 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022116140000000002 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287445+0000", 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2772000000000001e-05 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:417 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:32.711 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.427893126999997, 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028569959999999998, 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 417 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.712 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270459+0000", 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.035e-06 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.713 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271813+0000", 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001354037 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271834+0000", 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0302e-05 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271844+0000", 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9070000000000002e-06 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.714 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274939+0000", 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003095169 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275041+0000", 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000102045 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276123+0000", 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010819880000000001 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.715 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285162+0000", 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009039432 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.298984+0000", 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013821599 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299027+0000", 2024-09-15T02:49:32.716 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3445999999999997e-05 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:418 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~147456] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.426490626000003, 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027304571999999999, 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.717 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 418 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.718 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271861+0000", 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1829999999999999e-06 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272037+0000", 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017563799999999999 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272050+0000", 2024-09-15T02:49:32.719 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3506000000000001e-05 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272057+0000", 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999998e-06 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281725+0000", 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096680910000000002 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.720 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281887+0000", 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016169599999999999 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282772+0000", 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088466100000000002 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285439+0000", 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026669340000000001 2024-09-15T02:49:32.721 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299113+0000", 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013674009000000001 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299164+0000", 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1573999999999998e-05 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.722 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:419 3.0s0 3:038350ba:::smithi19125844-11:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.426285329000002, 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.031729234000000002, 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 419 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.723 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272066+0000", 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9000000000000005e-07 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.724 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272070+0000", 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1740000000000002e-06 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272078+0000", 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5329999999999997e-06 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272083+0000", 2024-09-15T02:49:32.725 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7859999999999996e-06 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284520+0000", 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012437190000000001 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284611+0000", 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0816000000000006e-05 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.726 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284926+0000", 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031568500000000001 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287991+0000", 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030649599999999998 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303704+0000", 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015713142999999999 2024-09-15T02:49:32.727 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303794+0000", 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9957000000000006e-05 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:420 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.426262321999999, 2024-09-15T02:49:32.728 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033404771999999999, 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 420 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.729 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272089+0000", 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.51e-07 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272091+0000", 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1730000000000002e-06 2024-09-15T02:49:32.730 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272096+0000", 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.904e-06 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272099+0000", 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2059999999999999e-06 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285286+0000", 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013187488000000001 2024-09-15T02:49:32.731 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285374+0000", 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7392999999999995e-05 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304237+0000", 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018863347999999999 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304386+0000", 2024-09-15T02:49:32.732 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014841 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.305493+0000", 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001107299 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:429 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.426188975000002, 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022788739999999998, 2024-09-15T02:49:32.733 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 429 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.734 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272162+0000", 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272166+0000", 2024-09-15T02:49:32.735 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9269999999999998e-06 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272174+0000", 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6569999999999997e-06 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272178+0000", 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6260000000000003e-06 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.736 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274127+0000", 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001949295 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274178+0000", 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0547999999999998e-05 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.294950+0000", 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020772112999999998 2024-09-15T02:49:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 50/33 ECSubReadReply(tid=60, attrs_read=0))", 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.420915706999999, 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017580023, 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.738 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0100000000000004e-07 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.739 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293753+0000", 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016317505999999999 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293859+0000", 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105707 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293862+0000", 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4819999999999999e-06 2024-09-15T02:49:32.740 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293910+0000", 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7763000000000003e-05 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.295015+0000", 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011048640000000001 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.741 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:457 3.9s0 3:9c49cf3d:::smithi19125844-14:head [append 0~0] snapc 11=[11,10,f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 36.450811911000002, 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017317169, 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 457 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.742 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247538+0000", 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247540+0000", 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1629999999999999e-06 2024-09-15T02:49:32.743 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247546+0000", 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9320000000000001e-06 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247563+0000", 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7268000000000001e-05 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.744 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247573+0000", 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8970000000000003e-06 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247766+0000", 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00019318499999999999 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.247773+0000", 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8600000000000004e-06 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.745 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254774+0000", 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070010150000000002 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.254891+0000", 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000117342 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.255604+0000", 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071258400000000005 2024-09-15T02:49:32.746 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.256332+0000", 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072847400000000001 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264776+0000", 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0084437450000000008 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.747 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:56.264856+0000", 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.9703999999999994e-05 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.12.22:2 3.9s0 3:94f52c52:::smithi19125844-20:head [assert-version v18,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 33.370494055999998, 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59485876100000001, 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.748 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327859+0000", 2024-09-15T02:49:32.749 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.446e-06 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327869+0000", 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7629999999999995e-06 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327898+0000", 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9054e-05 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.750 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327913+0000", 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4990999999999999e-05 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328005+0000", 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2015000000000006e-05 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328180+0000", 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017567599999999999 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.751 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.922715+0000", 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59453481600000002 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:630 3.9s0 3:9c49cf3d:::smithi19125844-14:head [copy-from ver 31] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 20.304439494, 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021438329999999998, 2024-09-15T02:49:32.752 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 630 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.753 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393915+0000", 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7570000000000002e-06 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393951+0000", 2024-09-15T02:49:32.754 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6347999999999998e-05 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393987+0000", 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6405999999999997e-05 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394003+0000", 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5956000000000001e-05 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394131+0000", 2024-09-15T02:49:32.755 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012749300000000001 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394155+0000", 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3754999999999999e-05 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401719+0000", 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075645340000000004 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.756 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401977+0000", 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000257543 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409969+0000", 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007991873 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414563+0000", 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045945830000000002 2024-09-15T02:49:32.757 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415199+0000", 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063606199999999996 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415349+0000", 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015002 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.758 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:631 3.9s0 3:9c49cf3d:::smithi19125844-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 20.304298555999999, 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017944699000000001, 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 631 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394053+0000", 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3179999999999999e-06 2024-09-15T02:49:32.760 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394059+0000", 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9120000000000003e-06 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394075+0000", 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5809999999999999e-05 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.761 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394085+0000", 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0586e-05 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394258+0000", 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017270999999999999 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394277+0000", 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9236999999999999e-05 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.762 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402333+0000", 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0080553299999999994 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402447+0000", 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00011408600000000001 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411771+0000", 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093238820000000004 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.763 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411860+0000", 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8857999999999995e-05 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411997+0000", 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013697000000000001 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 72/71 ECSubWriteReply(tid=134, last_complete=72'51, committed=1, applied=1))", 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.405508523000002, 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060209891000000001, 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.765 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292843+0000", 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3099999999999996e-07 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292848+0000", 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6729999999999997e-06 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352898+0000", 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060049335000000002 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352910+0000", 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.256e-05 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352988+0000", 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7831999999999996e-05 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.767 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.353052+0000", 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3559999999999995e-05 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:688 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.25947751, 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037318727000000003, 2024-09-15T02:49:32.768 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 688 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.769 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438875+0000", 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5809999999999999e-06 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438889+0000", 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3444999999999999e-05 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438927+0000", 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7842000000000001e-05 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438940+0000", 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3473e-05 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439086+0000", 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014553999999999999 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439109+0000", 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3283000000000001e-05 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448063+0000", 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089537490000000004 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448274+0000", 2024-09-15T02:49:32.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000210984 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476192+0000", 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027917830000000001 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:689 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.259388523, 2024-09-15T02:49:32.773 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037724385999999999, 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 689 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.774 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438963+0000", 2024-09-15T02:49:32.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5300000000000002e-07 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438968+0000", 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9180000000000002e-06 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438976+0000", 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6470000000000004e-06 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438981+0000", 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9130000000000002e-06 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439306+0000", 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032486700000000001 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439319+0000", 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2442999999999999e-05 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448872+0000", 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095532159999999998 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449323+0000", 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00045103599999999998 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476686+0000", 2024-09-15T02:49:32.778 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027363393 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:690 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 14.259306372999999, 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038145563, 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:32.779 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 690 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.780 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439045+0000", 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2200000000000003e-07 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439048+0000", 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.76e-06 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439064+0000", 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5336e-05 2024-09-15T02:49:32.781 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439069+0000", 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8849999999999998e-06 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439382+0000", 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031364799999999998 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439397+0000", 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4661000000000001e-05 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449673+0000", 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010276298 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.785 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449816+0000", 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014270799999999999 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.477190+0000", 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027373544999999999 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDScrubReserve(3.19s4 RELEASE e75)", 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:32.786 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.416315483, 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98297010699999998, 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.787 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282036+0000", 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2270000000000001e-06 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282044+0000", 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6720000000000004e-06 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264750+0000", 2024-09-15T02:49:32.788 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98270659599999999 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264763+0000", 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2466000000000001e-05 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264912+0000", 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000149524 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:32.789 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264954+0000", 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1461000000000002e-05 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.265005+0000", 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1161e-05 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s4 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.164037956, 2024-09-15T02:49:32.790 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018239887999999999, 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.791 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534314+0000", 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7549999999999999e-06 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534957+0000", 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00064235200000000001 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535041+0000", 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4050999999999999e-05 2024-09-15T02:49:32.792 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535056+0000", 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5588999999999999e-05 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535177+0000", 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120925 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535207+0000", 2024-09-15T02:49:32.793 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9864000000000001e-05 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552499+0000", 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017291890000000001 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552552+0000", 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3461999999999997e-05 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.794 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.bs4 77/76 [PushOp(3:d394ce14:::smithi19125844-2:head, version: 71'77, data_included: [262144~241664], data_size: 241664, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d394ce14:::smithi19125844-2:head@71'77, size: 2015232, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2015232, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 11.061904542000001, 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.24636945099999999, 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.795 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636447+0000", 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.637e-06 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636697+0000", 2024-09-15T02:49:32.796 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024920600000000003 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867157+0000", 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.23046036 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867176+0000", 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.929e-05 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.797 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867332+0000", 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015594099999999999 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.882815+0000", 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015483017 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.200863979999999, 2024-09-15T02:49:32.798 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025018460999999999, 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.799 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497487+0000", 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8000000000000002e-07 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499654+0000", 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00216738 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499711+0000", 2024-09-15T02:49:32.800 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7398999999999997e-05 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499723+0000", 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2019e-05 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499830+0000", 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000106718 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:32.801 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499850+0000", 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9732e-05 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522437+0000", 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022587494 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522505+0000", 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7439000000000003e-05 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.802 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.803 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:32.803 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:32.803 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:32.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:32.805 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:32.799+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:32.905 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:32.899+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:32.914 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:32.914 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:32.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:33.007 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:32.999+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:33.024 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.025 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:33.025 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:33.025 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:33.025 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:33.108 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:33.106+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:33.111 INFO:teuthology.orchestra.run.smithi152.stderr:Nothing to do for non-filestore backend 2024-09-15T02:49:33.136 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.563965512999999, 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.137 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:33.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:49:33.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:33.141 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.554023207, 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099046280000000004, 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:33.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579386+0000", 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.536e-06 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579616+0000", 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230366 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579660+0000", 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4489e-05 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579668+0000", 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4800000000000004e-06 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579731+0000", 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2733999999999999e-05 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579742+0000", 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589278+0000", 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095358219999999994 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589289+0000", 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0596e-05 2024-09-15T02:49:33.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 42.483018831000003, 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015141226000000001, 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:33.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650389+0000", 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650600+0000", 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021155999999999999 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650679+0000", 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8641999999999997e-05 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650693+0000", 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4294000000000001e-05 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650801+0000", 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108132 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650828+0000", 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6565999999999999e-05 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665503+0000", 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014675142 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665530+0000", 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6509999999999999e-05 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.440924596999999, 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011398508, 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.22e-07 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693204+0000", 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072156 2024-09-15T02:49:33.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693258+0000", 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3134999999999999e-05 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693266+0000", 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5259999999999993e-06 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693333+0000", 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6580000000000003e-05 2024-09-15T02:49:33.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693344+0000", 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0971999999999999e-05 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703854+0000", 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010510265 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703881+0000", 2024-09-15T02:49:33.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7248000000000001e-05 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.879093967999999, 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016572159999999999, 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254314+0000", 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1100000000000002e-07 2024-09-15T02:49:33.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255086+0000", 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077269100000000005 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255147+0000", 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1005999999999998e-05 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255166+0000", 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8581999999999999e-05 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255267+0000", 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100973 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255289+0000", 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2565000000000002e-05 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270855+0000", 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015565733999999999 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270885+0000", 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0298e-05 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:33.159 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.864603013, 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018797873, 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8999999999999998e-07 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269160+0000", 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035577800000000001 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3349000000000001e-05 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269211+0000", 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9439999999999999e-06 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269283+0000", 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2263999999999998e-05 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269296+0000", 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3145e-05 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287519+0000", 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018222955999999998 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287602+0000", 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3146999999999995e-05 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:33.164 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.856330387, 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010563233999999999, 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277078+0000", 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1500000000000004e-07 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278860+0000", 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017825339999999999 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278958+0000", 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7705000000000001e-05 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278971+0000", 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3481e-05 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279037+0000", 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5848999999999999e-05 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279045+0000", 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6359999999999994e-06 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287611+0000", 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085663470000000002 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287640+0000", 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8966999999999999e-05 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.168 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.205595942999999, 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010079322999999999, 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927813+0000", 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9630000000000001e-06 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928385+0000", 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057185600000000004 2024-09-15T02:49:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928464+0000", 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8615999999999994e-05 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928480+0000", 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6696000000000001e-05 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928602+0000", 2024-09-15T02:49:33.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121209 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928622+0000", 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.052e-05 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937813+0000", 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091904280000000005 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937891+0000", 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8034999999999994e-05 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(3.1s2 RELEASE e76)", 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.463772474000001, 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43008043499999998, 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:33.173 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669637+0000", 2024-09-15T02:49:33.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.917e-06 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669648+0000", 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1653e-05 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099516+0000", 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42986811200000002 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099534+0000", 2024-09-15T02:49:33.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7997000000000001e-05 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099653+0000", 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011822100000000001 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099681+0000", 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8496999999999998e-05 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099715+0000", 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4038000000000002e-05 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.649681490000001, 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012988501, 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483727+0000", 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2500000000000004e-07 2024-09-15T02:49:33.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483886+0000", 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000159028 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483939+0000", 2024-09-15T02:49:33.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3640999999999997e-05 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483950+0000", 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0462000000000001e-05 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.484771+0000", 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082086099999999999 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.496714+0000", 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011943684 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.631873166, 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010341001000000001, 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501535+0000", 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2309999999999999e-06 2024-09-15T02:49:33.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501651+0000", 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115538 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501708+0000", 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7479000000000003e-05 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501723+0000", 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4815e-05 2024-09-15T02:49:33.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501805+0000", 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2000000000000001e-05 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511875+0000", 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010069938 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:127fa005:::smithi19125844-18:head, version: 36'10, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:127fa005:::smithi19125844-18:head@36'10, size: 1212416, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.623613201, 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012919893999999999, 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509795+0000", 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8599999999999996e-07 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510077+0000", 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028238699999999999 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510142+0000", 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4518999999999997e-05 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510161+0000", 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8751999999999999e-05 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511930+0000", 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017696719999999999 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.522714+0000", 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010783578 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.598838705, 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0218164, 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.188 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:33.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6199999999999996e-07 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534737+0000", 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016761400000000001 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534787+0000", 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9836000000000002e-05 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534795+0000", 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1550000000000007e-06 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534859+0000", 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4162e-05 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534875+0000", 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6305000000000001e-05 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556345+0000", 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021469637999999999 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556385+0000", 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9827999999999999e-05 2024-09-15T02:49:33.192 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=251, reqid=client.4526.0:761, at_version=77'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.591043939, 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014142234, 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542364+0000", 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8700000000000005e-07 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542468+0000", 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010437099999999999 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542522+0000", 2024-09-15T02:49:33.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3295999999999999e-05 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542529+0000", 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1709999999999996e-06 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544564+0000", 2024-09-15T02:49:33.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00203476 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544579+0000", 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5605000000000001e-05 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556487+0000", 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011907687 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556506+0000", 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556999999999999e-05 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:18, version: 70'39, data_included: [262144~253952], data_size: 253952, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:18@70'39, size: 2064384, copy_subset: [], clone_subset: {}, snapset: 27=[]:{e=[b],18=[18],27=[27,26,25,24,21,1d]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2064384, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.435185052, 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010750387, 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698224+0000", 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2570000000000002e-06 2024-09-15T02:49:33.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698584+0000", 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035969000000000002 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698700+0000", 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116018 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698723+0000", 2024-09-15T02:49:33.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3173e-05 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698830+0000", 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010714599999999999 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.708973+0000", 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010142103 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 81/76 ECSubRead(tid=114, to_read={3:517ce0e6:::smithi19125844-47:head=0,475136,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 7.6442975440000005, 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69833002899999996, 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489111+0000", 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7269999999999999e-06 2024-09-15T02:49:33.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489134+0000", 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3033000000000001e-05 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187052+0000", 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69791777600000005 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187066+0000", 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3567000000000001e-05 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187169+0000", 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010303599999999999 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187440+0000", 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027088999999999998 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "replica_scrub(pg: 3.0s3,from:0'0,to:85'34,epoch:86/71,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.7159571339999999, 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98343883300000001, 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9999999999999999e-07 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417455+0000", 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-06 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400723+0000", 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98326817399999999 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400733+0000", 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5929999999999995e-06 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400839+0000", 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106413 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400855+0000", 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5150000000000001e-05 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400889+0000", 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4353000000000001e-05 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=173, reqid=client.4526.0:901, at_version=88'59, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:33.212 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.85257247300000005, 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019105225999999999, 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281129+0000", 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029418700000000001 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281252+0000", 2024-09-15T02:49:33.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123051 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281268+0000", 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5682999999999999e-05 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281408+0000", 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013955900000000001 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281431+0000", 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2952e-05 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299857+0000", 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018425891 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299940+0000", 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3423000000000004e-05 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=175, reqid=client.4526.0:903, at_version=88'61, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.83128105200000002, 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011439048, 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302127+0000", 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.119e-06 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302328+0000", 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000200834 2024-09-15T02:49:33.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302386+0000", 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7895e-05 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302394+0000", 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0770000000000008e-06 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302465+0000", 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1017000000000001e-05 2024-09-15T02:49:33.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302481+0000", 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5599000000000001e-05 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313526+0000", 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011045743 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313565+0000", 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8763999999999997e-05 2024-09-15T02:49:33.220 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=178, reqid=client.4526.0:905, at_version=0'0, trim_to=0'0, roll_forward_to=88'63))", 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.81366569499999997, 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15873490300000001, 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319766+0000", 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3816e-05 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476980+0000", 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.157214144 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476993+0000", 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2286000000000001e-05 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477134+0000", 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141006 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477167+0000", 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3847000000000003e-05 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478412+0000", 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012443930000000001 2024-09-15T02:49:33.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478476+0000", 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4579999999999995e-05 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.225 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:33.227 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:33.224+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:33.246 INFO:tasks.rados.rados.0.smithi191.stdout:356: done (2 left) 2024-09-15T02:49:33.246 INFO:tasks.rados.rados.0.smithi191.stdout:357: done (1 left) 2024-09-15T02:49:33.246 INFO:tasks.rados.rados.0.smithi191.stdout:358: done (0 left) 2024-09-15T02:49:33.246 INFO:tasks.rados.rados.0.smithi191.stdout:359: read oid 45 snap -1 2024-09-15T02:49:33.247 INFO:tasks.rados.rados.0.smithi191.stdout:359: expect deleted 2024-09-15T02:49:33.247 INFO:tasks.rados.rados.0.smithi191.stdout:360: copy_from oid 23 from oid 32 current snap is 34 2024-09-15T02:49:33.247 INFO:tasks.rados.rados.0.smithi191.stdout:361: snap_create 2024-09-15T02:49:33.248 INFO:tasks.rados.rados.0.smithi191.stdout:360: finishing copy_from to smithi19125844-23 2024-09-15T02:49:33.248 INFO:tasks.rados.rados.0.smithi191.stdout:360: got expected ENOENT (src dne) 2024-09-15T02:49:33.249 INFO:tasks.rados.rados.0.smithi191.stdout:360: finishing copy_from racing read to smithi19125844-23 2024-09-15T02:49:33.327 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:33.319+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:33.332 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.332 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:33.332 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:33.332 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:33.428 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:33.423+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:33.442 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.443 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:33.443 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:33.443 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:33.443 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:33.530 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:33.523+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.98088396, 2024-09-15T02:49:33.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:49:33.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.968367774999997, 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:33.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:49:33.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.967328615, 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.563 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:49:33.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 42.899577608999998, 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015031548, 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651048+0000", 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4900000000000001e-07 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651266+0000", 2024-09-15T02:49:33.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000218233 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651324+0000", 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8544000000000001e-05 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651335+0000", 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336000000000001e-05 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651408+0000", 2024-09-15T02:49:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3415000000000001e-05 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651424+0000", 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6001000000000001e-05 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666045+0000", 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014620861000000001 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666079+0000", 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3809000000000001e-05 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.858205224999999, 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015570855999999999, 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692736+0000", 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031561799999999999 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692809+0000", 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3174000000000001e-05 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692821+0000", 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1627000000000001e-05 2024-09-15T02:49:33.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692906+0000", 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5172000000000004e-05 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692920+0000", 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4361e-05 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707954+0000", 2024-09-15T02:49:33.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015033451999999999 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707991+0000", 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6714000000000001e-05 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.801143070000002, 2024-09-15T02:49:33.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014530774999999999, 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749483+0000", 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4799999999999997e-07 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749717+0000", 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000234408 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749768+0000", 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1155000000000001e-05 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749777+0000", 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1609999999999992e-06 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749853+0000", 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5803000000000006e-05 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749872+0000", 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556e-05 2024-09-15T02:49:33.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.763971+0000", 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014098934 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.764013+0000", 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1810000000000001e-05 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.580 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.296262644000002, 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012056918, 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254363+0000", 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7199999999999997e-07 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254896+0000", 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053267899999999999 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255034+0000", 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013821100000000001 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255066+0000", 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255135+0000", 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8885000000000002e-05 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255148+0000", 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2498e-05 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266383+0000", 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011235187000000001 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266419+0000", 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6358e-05 2024-09-15T02:49:33.584 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.259488038999997, 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011905962000000001, 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291138+0000", 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291342+0000", 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203403 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291572+0000", 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230084 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291579+0000", 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6990000000000003e-06 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291641+0000", 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2093000000000003e-05 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291650+0000", 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303018+0000", 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011368296 2024-09-15T02:49:33.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303043+0000", 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4901000000000001e-05 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.259253649000001, 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011692335, 2024-09-15T02:49:33.589 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291548+0000", 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000177021 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291673+0000", 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012469299999999999 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291678+0000", 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296939+0000", 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052609120000000004 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296948+0000", 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303051+0000", 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006102433 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303064+0000", 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2948999999999999e-05 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.593 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/33 ECSubWrite(tid=212, reqid=client.4526.0:450, at_version=52'22, trim_to=0'0, roll_forward_to=49'21))", 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 37.302165465999998, 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015878961, 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248460+0000", 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249338+0000", 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087768999999999998 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249433+0000", 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5333000000000004e-05 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249451+0000", 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8406e-05 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249656+0000", 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020463100000000001 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249685+0000", 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8943000000000001e-05 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264308+0000", 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014623486999999999 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264338+0000", 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0020999999999999e-05 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.597 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 62/33 ECSubWrite(tid=307, reqid=osd.13.0:307, at_version=62'49, trim_to=0'0, roll_forward_to=61'47))", 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.212890869999999, 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011170724999999999, 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:33.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337734+0000", 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337735+0000", 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0299999999999999e-07 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.337799+0000", 2024-09-15T02:49:33.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4206000000000002e-05 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338542+0000", 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074343499999999997 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338565+0000", 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2931000000000001e-05 2024-09-15T02:49:33.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338649+0000", 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4103000000000004e-05 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.338667+0000", 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7419999999999999e-05 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.348862+0000", 2024-09-15T02:49:33.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010195651 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:06.348905+0000", 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2376000000000002e-05 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.26396772, 2024-09-15T02:49:33.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020897651, 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286658+0000", 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.58e-07 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287075+0000", 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041690100000000001 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287318+0000", 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024316499999999999 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287324+0000", 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9580000000000004e-06 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287521+0000", 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197746 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287532+0000", 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366e-05 2024-09-15T02:49:33.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307520+0000", 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019987953999999999 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307555+0000", 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5102999999999999e-05 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.263530968000001, 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020477789999999999, 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3200000000000001e-07 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287283+0000", 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018875200000000001 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287367+0000", 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4036000000000007e-05 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287374+0000", 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5340000000000002e-06 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295137+0000", 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077628599999999999 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295149+0000", 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307560+0000", 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012411447000000001 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307572+0000", 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-15T02:49:33.610 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=413, reqid=client.4526.0:673, at_version=72'80, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.257688140999999, 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015088048, 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8799999999999999e-07 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292992+0000", 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5479000000000002e-05 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293291+0000", 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029850500000000002 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293295+0000", 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4179999999999999e-06 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302806+0000", 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095105680000000005 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302817+0000", 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1464999999999999e-05 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307993+0000", 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051757540000000003 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308025+0000", 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1671000000000002e-05 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:c, version: 71'82, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:c@71'82, size: 1081344, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.615 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.83593945, 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013023088, 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:33.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714687+0000", 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.866e-06 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715107+0000", 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419293 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715195+0000", 2024-09-15T02:49:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8758999999999995e-05 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715215+0000", 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0143e-05 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715370+0000", 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000154611 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.727709+0000", 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012338416 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:head, version: 71'83, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:head@71'83, size: 2293760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.56510256, 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011824942, 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985524+0000", 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.533e-06 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985735+0000", 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021152599999999999 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985816+0000", 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0516e-05 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985833+0000", 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7413000000000001e-05 2024-09-15T02:49:33.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985974+0000", 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141161 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.997347+0000", 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011372793000000001 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7f1f21a6:::smithi19125844-10:head, version: 71'85, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:7f1f21a6:::smithi19125844-10:head@71'85, size: 1736704, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.306006011999999, 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014391049, 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244620+0000", 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4300000000000001e-06 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245318+0000", 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069816099999999999 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245404+0000", 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5782e-05 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245423+0000", 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8272999999999999e-05 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245544+0000", 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121785 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.259010+0000", 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013465618 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=472, reqid=client.4526.0:793, at_version=79'84, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.067157823000001, 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013978862999999999, 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.626 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483468+0000", 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483595+0000", 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012675800000000001 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483644+0000", 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9200000000000003e-05 2024-09-15T02:49:33.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483653+0000", 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4030000000000007e-06 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483747+0000", 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4239000000000004e-05 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483762+0000", 2024-09-15T02:49:33.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5698000000000001e-05 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497313+0000", 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013550859 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497446+0000", 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013259400000000001 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=473, reqid=client.4526.0:794, at_version=79'85, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.060942594, 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014861180999999999, 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489684+0000", 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.282e-06 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490105+0000", 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042170800000000002 2024-09-15T02:49:33.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490163+0000", 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8158000000000001e-05 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490173+0000", 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4350000000000003e-06 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490301+0000", 2024-09-15T02:49:33.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127928 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490314+0000", 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3104e-05 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504495+0000", 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014181545 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504543+0000", 2024-09-15T02:49:33.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8021e-05 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s3 88/76 ECSubWrite(tid=173, reqid=client.4526.0:901, at_version=88'59, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.281302+0000", 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.269322598, 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011197344, 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281302+0000", 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281302+0000", 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281303+0000", 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0699999999999996e-07 2024-09-15T02:49:33.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281448+0000", 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000144327 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281563+0000", 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011501 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281624+0000", 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0974000000000003e-05 2024-09-15T02:49:33.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281769+0000", 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014515400000000001 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281794+0000", 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5360000000000001e-05 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:33.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.292461+0000", 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010666867 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.292500+0000", 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8745000000000003e-05 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.639 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:33.641 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:33.636+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:33.663 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.6 2024-09-15T02:49:33.663 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-6 show-label 2024-09-15T02:49:33.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:33.740 INFO:teuthology.orchestra.run.smithi152.stdout:inferring bluefs devices from bluestore path 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout: "/var/lib/ceph/osd/ceph-6/block": { 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_uuid": "7f8ecd2a-7829-496b-b349-5fdebbb6962a", 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 96636764160, 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout: "btime": "2024-09-15T02:47:38.088942+0000", 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "main", 2024-09-15T02:49:33.741 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_blocks": "23592960", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_blocks_per_key": "128", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_bytes_per_block": "4096", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "bfm_size": "96636764160", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs": "1", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_fsid": "24e12693-a4e4-4f9a-bb1d-999f41be75a2", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "created_at": "2024-09-15T02:47:40.523378Z", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "kv_backend": "rocksdb", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "magic": "ceph osd volume v026", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "mkfs_done": "yes", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "ready": "ready", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "require_osd_release": "17", 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: "whoami": "6" 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:33.742 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:33.743 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:33.736+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:33.744 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.6 2024-09-15T02:49:33.744 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-6 show-sharding 2024-09-15T02:49:33.748 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.748 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:33.748 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:33.748 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:33.842 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:33.835+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:33.854 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:33.854 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:33.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:33.871 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:33.902 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:33.933 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:33.942 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:33.938+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:33.963 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:33.963 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:33.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:33.963 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:33.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.963 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.597362727999993, 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:33.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:33.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:33.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:33.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.237791213999998, 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:33.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:33.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.976 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.629138073000007, 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026584729000000001, 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75 2024-09-15T02:49:33.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331970+0000", 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.331971+0000", 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332135+0000", 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000164098 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332158+0000", 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332165+0000", 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8669999999999996e-06 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344120+0000", 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011955222 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344205+0000", 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4895000000000006e-05 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.344336+0000", 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130844 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.349645+0000", 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053089770000000003 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358356+0000", 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087112079999999998 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358462+0000", 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105664 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358555+0000", 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3110999999999997e-05 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.628928388000006, 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.392234770000002, 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.988 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.374716194000001, 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:33.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:33.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 66.560853766999998, 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:33.998 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:33.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:34.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:34.004 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.254610129, 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:34.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:34.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:34.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:34.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:34.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:34.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.254552296, 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:34.013 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:34.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:34.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.254517110000002, 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:34.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:34.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 35.519212551000003, 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.027 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.680780337999998, 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:670 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.677486475999999, 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027236222000000001, 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 670 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8800000000000003e-07 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283625+0000", 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3329999999999999e-06 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283630+0000", 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7450000000000001e-06 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283638+0000", 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7349999999999996e-06 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283962+0000", 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032339899999999998 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284074+0000", 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011208499999999999 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298432+0000", 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014357939 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298517+0000", 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310858+0000", 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012341388999999999 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.bs2 72/33 ECSubRead(tid=420, to_read={3:d2661d0e:::smithi19125844-43:head=0,4096,0}, subchunks={3:d2661d0e:::smithi19125844-43:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.650305137, 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085265647999999999, 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310804+0000", 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310814+0000", 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0115000000000001e-05 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395542+0000", 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.084728263999999998 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395552+0000", 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5899999999999997e-06 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395608+0000", 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5832999999999999e-05 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.396069+0000", 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046113399999999998 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:753 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.500356664, 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027324345, 2024-09-15T02:49:34.040 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 753 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460754+0000", 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9709999999999998e-06 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460760+0000", 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7510000000000002e-06 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460773+0000", 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2136999999999999e-05 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460780+0000", 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471867+0000", 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011087001000000001 2024-09-15T02:49:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471873+0000", 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000002e-06 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.484980+0000", 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0131073370000001 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485369+0000", 2024-09-15T02:49:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038873500000000002 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488076+0000", 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027074099999999999 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:754 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.500321076000001, 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0276021209999999, 2024-09-15T02:49:34.045 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 754 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460788+0000", 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8899999999999999e-07 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460791+0000", 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.411e-06 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460798+0000", 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4679999999999997e-06 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460802+0000", 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2889999999999999e-06 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471896+0000", 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011093591 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471897+0000", 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.331e-06 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485874+0000", 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0139766539999999 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485962+0000", 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8481999999999998e-05 2024-09-15T02:49:34.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488389+0000", 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024273060000000002 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:755 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.500302058999999, 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0311918470000001, 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.050 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 755 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460807+0000", 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460809+0000", 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460815+0000", 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1249999999999998e-06 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460819+0000", 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9670000000000003e-06 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471918+0000", 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098856000000001 2024-09-15T02:49:34.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471919+0000", 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488503+0000", 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0165840150000001 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488618+0000", 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011474199999999999 2024-09-15T02:49:34.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491998+0000", 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033803650000000002 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:766 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.500285030000001, 2024-09-15T02:49:34.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0356599440000001, 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 766 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460824+0000", 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.27e-07 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460831+0000", 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-09-15T02:49:34.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460837+0000", 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460841+0000", 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5889999999999999e-06 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471939+0000", 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098591 2024-09-15T02:49:34.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471940+0000", 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488932+0000", 2024-09-15T02:49:34.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0169917230000001 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.489075+0000", 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014318299999999999 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.496483+0000", 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074080589999999998 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:767 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.500263971999999, 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.037162342, 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 767 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460845+0000", 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3200000000000001e-07 2024-09-15T02:49:34.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460847+0000", 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.554e-06 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460853+0000", 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020000000000002e-06 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460857+0000", 2024-09-15T02:49:34.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0679999999999998e-06 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471961+0000", 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011103755999999999 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471962+0000", 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3600000000000003e-07 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.490940+0000", 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.018977662 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491003+0000", 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3081000000000001e-05 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.498007+0000", 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070041510000000001 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.065 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:768 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.500247054000001, 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.038324754, 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 768 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.066 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460862+0000", 2024-09-15T02:49:34.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.34e-07 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460864+0000", 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6639999999999998e-06 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460870+0000", 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0859999999999998e-06 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:34.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460874+0000", 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.563e-06 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471982+0000", 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011108122999999999 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471983+0000", 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6700000000000003e-07 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:34.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492584+0000", 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0206010889999999 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492705+0000", 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012115100000000001 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.499186+0000", 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006481077 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.071 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:34.071 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:34.071 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:34.072 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:34.072 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:34.073 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:34.073 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:34.073 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:34.067+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:34.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:34.078 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:34.108 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:34.137 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:34.166 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:34.173 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:34.172+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:34.186 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:34.186 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:34.186 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:34.186 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:34.194 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:34.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:34.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:34.239 INFO:tasks.rados.rados.0.smithi191.stdout:359: done (2 left) 2024-09-15T02:49:34.240 INFO:tasks.rados.rados.0.smithi191.stdout:360: done (1 left) 2024-09-15T02:49:34.240 INFO:tasks.rados.rados.0.smithi191.stdout:361: done (0 left) 2024-09-15T02:49:34.240 INFO:tasks.rados.rados.0.smithi191.stdout:362: read oid 15 snap -1 2024-09-15T02:49:34.240 INFO:tasks.rados.rados.0.smithi191.stdout:362: expect deleted 2024-09-15T02:49:34.240 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 18 current snap is 35 2024-09-15T02:49:34.240 INFO:tasks.rados.rados.0.smithi191.stdout:363: seq_num 132 ranges {0=671744,671744=573440,1245184=753664,1998848=229376} 2024-09-15T02:49:34.248 INFO:tasks.rados.rados.0.smithi191.stdout:363: writing smithi19125844-18 from 0 to 671744 tid 1 2024-09-15T02:49:34.250 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:34.253 INFO:tasks.rados.rados.0.smithi191.stdout:363: writing smithi19125844-18 from 671744 to 1245184 tid 2 2024-09-15T02:49:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:34.251+0000 7fb6327fc700 0 -- 172.21.15.6:0/2605925383 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fb634193580 msgr2=0x7fb634195960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:34.260 INFO:tasks.rados.rados.0.smithi191.stdout:363: writing smithi19125844-18 from 1245184 to 1998848 tid 3 2024-09-15T02:49:34.262 INFO:tasks.rados.rados.0.smithi191.stdout:363: writing smithi19125844-18 from 1998848 to 2228224 tid 4 2024-09-15T02:49:34.262 INFO:tasks.rados.rados.0.smithi191.stdout:364: setattr oid 41 current snap is 35 2024-09-15T02:49:34.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:49:34.265 INFO:tasks.rados.rados.0.smithi191.stdout:362: done (2 left) 2024-09-15T02:49:34.265 INFO:tasks.rados.rados.0.smithi191.stdout:365: rollback oid 26 current snap is 35 2024-09-15T02:49:34.265 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 26 to 28 2024-09-15T02:49:34.265 INFO:tasks.rados.rados.0.smithi191.stdout:366: snap_create 2024-09-15T02:49:34.273 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:34.270+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:34.275 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 41 v 117 (ObjNum 4 snap 0 seq_num 304) dirty exists 2024-09-15T02:49:34.275 INFO:tasks.rados.rados.0.smithi191.stdout:365: finishing rollback tid 0 to smithi19125844-26 2024-09-15T02:49:34.278 INFO:tasks.rados.rados.0.smithi191.stdout:363: finishing write tid 1 to smithi19125844-18 2024-09-15T02:49:34.279 INFO:tasks.rados.rados.0.smithi191.stdout:365: finishing rollback tid 1 to smithi19125844-26 2024-09-15T02:49:34.279 INFO:tasks.rados.rados.0.smithi191.stdout:365: finishing rollback tid 2 to smithi19125844-26 2024-09-15T02:49:34.279 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 26 v 82 (ObjNum 111 snap 22 seq_num 111) dirty exists 2024-09-15T02:49:34.283 INFO:tasks.rados.rados.0.smithi191.stdout:363: finishing write tid 2 to smithi19125844-18 2024-09-15T02:49:34.284 INFO:tasks.rados.rados.0.smithi191.stdout:363: finishing write tid 3 to smithi19125844-18 2024-09-15T02:49:34.287 INFO:tasks.rados.rados.0.smithi191.stdout:363: finishing write tid 4 to smithi19125844-18 2024-09-15T02:49:34.290 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:34.290 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:34.291 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:34.292 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:34.292 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:34.294 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:34.311 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:34.311 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:34.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:34.374 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:34.367+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:34.468 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:34.468 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:34.475 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:34.472+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:34.478 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:34.576 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:34.572+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:34.587 INFO:teuthology.orchestra.run.smithi152.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P 2024-09-15T02:49:34.591 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:34.591 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:34.593 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.6 2024-09-15T02:49:34.593 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-15T02:49:34.594 INFO:tasks.thrashosds.thrasher:applying new sharding: m(4) P(2,0-) L(2) 2024-09-15T02:49:34.594 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-6 --sharding 'm(4) P(2,0-) L(2)' reshard 2024-09-15T02:49:34.601 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:34.651 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:49:34.646+0000 7f8adf9ac840 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:49:34.677 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:34.674+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:34.699 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:34.699 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:34.708 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:34.778 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:34.772+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:34.803 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:34.803 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:34.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:34.879 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:34.872+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:34.926 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:34.926 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:34.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:34.980 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:34.976+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:35.035 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.036 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:35.063 INFO:teuthology.orchestra.run.smithi152.stdout:reshard success 2024-09-15T02:49:35.073 INFO:tasks.rados.rados.0.smithi191.stdout:363: finishing write tid 5 to smithi19125844-18 2024-09-15T02:49:35.078 INFO:tasks.rados.rados.0.smithi191.stdout:363: finishing write tid 6 to smithi19125844-18 2024-09-15T02:49:35.078 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 18 v 61 (ObjNum 132 snap 35 seq_num 132) dirty exists 2024-09-15T02:49:35.078 INFO:tasks.rados.rados.0.smithi191.stdout:363: left oid 18 (ObjNum 132 snap 35 seq_num 132) 2024-09-15T02:49:35.080 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:35.075+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:35.145 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.145 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:35.181 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:35.175+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:35.245 INFO:tasks.rados.rados.0.smithi191.stdout:363: done (3 left) 2024-09-15T02:49:35.245 INFO:tasks.rados.rados.0.smithi191.stdout:364: done (2 left) 2024-09-15T02:49:35.245 INFO:tasks.rados.rados.0.smithi191.stdout:365: done (1 left) 2024-09-15T02:49:35.245 INFO:tasks.rados.rados.0.smithi191.stdout:366: done (0 left) 2024-09-15T02:49:35.245 INFO:tasks.rados.rados.0.smithi191.stdout:367: delete oid 45 current snap is 36 2024-09-15T02:49:35.246 INFO:tasks.rados.rados.0.smithi191.stdout:367: done (0 left) 2024-09-15T02:49:35.246 INFO:tasks.rados.rados.0.smithi191.stdout:368: delete oid 37 current snap is 36 2024-09-15T02:49:35.251 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:49:35.252 INFO:tasks.rados.rados.0.smithi191.stdout:368: done (0 left) 2024-09-15T02:49:35.252 INFO:tasks.rados.rados.0.smithi191.stdout:369: snap_create 2024-09-15T02:49:35.266 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.266 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:35.320 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:35.315+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:35.383 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:35.378+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:35.423 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.423 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.426 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.6 2024-09-15T02:49:35.426 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-6 fsck 2024-09-15T02:49:35.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:35.484 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:35.480+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:35.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-15T02:49:35.522+0000 7fd9ce98e840 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:49:35.531 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.531 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:35.586 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:35.580+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:35.640 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.640 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:35.687 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:35.684+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:35.748 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.748 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:35.787 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:35.779+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:35.862 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:35.862 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:35.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:35.916 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:35.911+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:35.989 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:35.988+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:36.019 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:36.019 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:36.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:36.090 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:36.088+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:36.127 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:36.127 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:36.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:36.191 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:36.188+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:36.235 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:36.236 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:36.244 INFO:tasks.rados.rados.0.smithi191.stdout:369: done (0 left) 2024-09-15T02:49:36.244 INFO:tasks.rados.rados.0.smithi191.stdout:370: read oid 48 snap -1 2024-09-15T02:49:36.245 INFO:tasks.rados.rados.0.smithi191.stdout:370: expect deleted 2024-09-15T02:49:36.245 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 14 current snap is 37 2024-09-15T02:49:36.245 INFO:tasks.rados.rados.0.smithi191.stdout:371: seq_num 135 ranges {98304=720896,819200=622592,1441792=688128,2129920=245760} 2024-09-15T02:49:36.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:36.255 INFO:tasks.rados.rados.0.smithi191.stdout:371: writing smithi19125844-14 from 98304 to 819200 tid 1 2024-09-15T02:49:36.258 INFO:tasks.rados.rados.0.smithi191.stdout:371: writing smithi19125844-14 from 819200 to 1441792 tid 2 2024-09-15T02:49:36.262 INFO:tasks.rados.rados.0.smithi191.stdout:371: writing smithi19125844-14 from 1441792 to 2129920 tid 3 2024-09-15T02:49:36.263 INFO:tasks.rados.rados.0.smithi191.stdout:371: writing smithi19125844-14 from 2129920 to 2375680 tid 4 2024-09-15T02:49:36.263 INFO:tasks.rados.rados.0.smithi191.stdout:372: delete oid 49 current snap is 37 2024-09-15T02:49:36.276 INFO:tasks.rados.rados.0.smithi191.stdout:370: done (2 left) 2024-09-15T02:49:36.277 INFO:tasks.rados.rados.0.smithi191.stdout:372: done (1 left) 2024-09-15T02:49:36.277 INFO:tasks.rados.rados.0.smithi191.stdout:373: snap_remove snap 36 2024-09-15T02:49:36.292 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:36.288+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:36.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:36.299 INFO:teuthology.orchestra.run.smithi152.stdout:fsck success 2024-09-15T02:49:36.309 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-15T02:49:36.310 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-15T02:49:36.310 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-15T02:49:36.310 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-15T02:49:36.312 INFO:tasks.ceph.osd.6:Started 2024-09-15T02:49:36.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:36.367 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:36.403 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:36.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:36.405 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:36.400+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:config diff 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:config diff get 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:config get 2024-09-15T02:49:36.412 INFO:teuthology.orchestra.run.smithi152.stderr:config help [] 2024-09-15T02:49:36.413 INFO:teuthology.orchestra.run.smithi152.stderr:config set ... 2024-09-15T02:49:36.413 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-15T02:49:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T02:49:36.414 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-15T02:49:36.447 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:36.477 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:36.506 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:36.500+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:36.506 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:36.506 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:36.506 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:36.506 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:36.506 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:36.506 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:36.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:36.535 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:36.563 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:36.591 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:36.606 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:36.602+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:36.620 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:36.620 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:36.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.983255861000004, 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016823131000000002, 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:36.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634134+0000", 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634136+0000", 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7940000000000001e-06 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634475+0000", 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033968099999999999 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634663+0000", 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018762999999999999 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634681+0000", 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7974000000000001e-05 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634905+0000", 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022468600000000001 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634923+0000", 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7254000000000001e-05 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650923+0000", 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015999777999999999 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.650957+0000", 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4334e-05 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 49/33 ECSubWrite(tid=155, reqid=client.4526.0:355, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:36.625 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.942181736999999, 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019850428999999999, 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675208+0000", 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9999999999999997e-07 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675436+0000", 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227369 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675628+0000", 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019193300000000001 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.675646+0000", 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7830999999999999e-05 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.676067+0000", 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042108899999999998 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.676096+0000", 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9149000000000002e-05 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695024+0000", 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018927979000000001 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695058+0000", 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4479e-05 2024-09-15T02:49:36.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 49/33 ECSubWrite(tid=156, reqid=client.4526.0:356, at_version=49'19, trim_to=0'0, roll_forward_to=48'16))", 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.933841219999998, 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011525719, 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683548+0000", 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.683550+0000", 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.03e-06 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684004+0000", 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045311 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684167+0000", 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163869 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684188+0000", 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0190000000000002e-05 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684753+0000", 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056561700000000003 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.684768+0000", 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4807000000000001e-05 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695067+0000", 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010299341 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.695074+0000", 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7549999999999997e-06 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.867798798999999, 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016121677000000001, 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.634 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749591+0000", 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749593+0000", 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.708e-06 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749692+0000", 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9852000000000006e-05 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749755+0000", 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2292999999999995e-05 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749767+0000", 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2646000000000001e-05 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749863+0000", 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5872999999999995e-05 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.749883+0000", 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0329000000000001e-05 2024-09-15T02:49:36.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.765618+0000", 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015734416000000001 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.765712+0000", 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4560000000000003e-05 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.327330273000001, 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016610296, 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290059+0000", 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290061+0000", 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.327e-06 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290153+0000", 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2641999999999998e-05 2024-09-15T02:49:36.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290291+0000", 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013733500000000001 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290300+0000", 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0459999999999994e-06 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290359+0000", 2024-09-15T02:49:36.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9695999999999999e-05 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290374+0000", 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4831000000000001e-05 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306642+0000", 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016267321000000001 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306670+0000", 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8098000000000001e-05 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.327217208999997, 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016516073999999999, 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290172+0000", 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290173+0000", 2024-09-15T02:49:36.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5400000000000002e-07 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290241+0000", 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7681000000000005e-05 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290326+0000", 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5348999999999999e-05 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290330+0000", 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.498e-06 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297821+0000", 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074905040000000003 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.297832+0000", 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0905e-05 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306676+0000", 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088443059999999997 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.306688+0000", 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2377e-05 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=61, reqid=client.4526.0:512, at_version=60'24, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:36.647 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.248578+0000", 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.368811432999998, 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022275555999999998, 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.248578+0000", 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.248578+0000", 2024-09-15T02:49:36.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.248579+0000", 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.06e-07 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249504+0000", 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000925583 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249591+0000", 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7118000000000002e-05 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249614+0000", 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2520000000000001e-05 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249716+0000", 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010254 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.249727+0000", 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0754000000000001e-05 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270830+0000", 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021102889 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270854+0000", 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3646e-05 2024-09-15T02:49:36.651 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=62, reqid=client.4526.0:513, at_version=60'25, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.256424+0000", 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.360966083000001, 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014458907, 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256424+0000", 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256424+0000", 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256425+0000", 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4139999999999999e-06 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256529+0000", 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010399999999999999 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256595+0000", 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5988000000000006e-05 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.256607+0000", 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1613e-05 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.260325+0000", 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003718341 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.260341+0000", 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6597000000000001e-05 2024-09-15T02:49:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270869+0000", 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010527549000000001 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.270882+0000", 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3405e-05 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/53 ECSubWrite(tid=63, reqid=client.4526.0:514, at_version=60'26, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.259676+0000", 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.357714021, 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011625017, 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259676+0000", 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259676+0000", 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259677+0000", 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259809+0000", 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013223600000000001 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259862+0000", 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2568999999999998e-05 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4239999999999993e-06 2024-09-15T02:49:36.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.266670+0000", 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068003999999999998 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.266685+0000", 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4861000000000001e-05 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271278+0000", 2024-09-15T02:49:36.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045931949999999996 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.271301+0000", 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2104e-05 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 67/53 ECSubWrite(tid=84, reqid=osd.0.0:84, at_version=67'43, trim_to=0'0, roll_forward_to=62'39))", 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:11.397299+0000", 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.220090279000001, 2024-09-15T02:49:36.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.38772314400000002, 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.397299+0000", 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.397299+0000", 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.397300+0000", 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.15e-06 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.397334+0000", 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3621000000000002e-05 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.783348+0000", 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.386014197 2024-09-15T02:49:36.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.783360+0000", 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2109e-05 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.783510+0000", 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014970399999999999 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.783532+0000", 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1515999999999999e-05 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.784961+0000", 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001429385 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:11.785022+0000", 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1462000000000002e-05 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:16.276706+0000", 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.340683894000001, 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018462817999999999, 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276706+0000", 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276706+0000", 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276706+0000", 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.99e-07 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276901+0000", 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019480800000000001 2024-09-15T02:49:36.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.280456+0000", 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035550669999999999 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.280468+0000", 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2478e-05 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.280536+0000", 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7293999999999996e-05 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.280558+0000", 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1997e-05 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.295152+0000", 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014594223 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.295169+0000", 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6752000000000002e-05 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 72/71 ECSubWrite(tid=406, reqid=osd.13.0:406, at_version=72'100, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:16.276907+0000", 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.340483067000001, 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015748945, 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276907+0000", 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276907+0000", 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276907+0000", 2024-09-15T02:49:36.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2199999999999999e-07 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.276923+0000", 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5670000000000001e-05 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284218+0000", 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072953779999999999 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284230+0000", 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1592e-05 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284325+0000", 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5583999999999998e-05 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284342+0000", 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6538999999999999e-05 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.292635+0000", 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082937250000000001 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.292655+0000", 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0035e-05 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:36.674 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:16.284474+0000", 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.332915271000001, 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011180681, 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284474+0000", 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284474+0000", 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284475+0000", 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2000000000000001e-07 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284636+0000", 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016114499999999999 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.285020+0000", 2024-09-15T02:49:36.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038440300000000001 2024-09-15T02:49:36.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.285027+0000", 2024-09-15T02:49:36.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4910000000000002e-06 2024-09-15T02:49:36.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.288021+0000", 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029944339999999998 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.288036+0000", 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5472e-05 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.295433+0000", 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073963120000000004 2024-09-15T02:49:36.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.295655+0000", 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022220399999999999 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.13s2 72/71 ECSubWrite(tid=98, reqid=osd.15.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=72'84))", 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:16.284649+0000", 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.332741024000001, 2024-09-15T02:49:36.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011232928999999999, 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284649+0000", 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284649+0000", 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284649+0000", 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5099999999999998e-07 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.284658+0000", 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6980000000000004e-06 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.285070+0000", 2024-09-15T02:49:36.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041219100000000001 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.285076+0000", 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.383e-06 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.294916+0000", 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098407849999999995 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.294943+0000", 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6797000000000001e-05 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.295865+0000", 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092131500000000002 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:16.295881+0000", 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6909e-05 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.686 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs5 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.497599+0000", 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.119790435000001, 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021170299, 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497599+0000", 2024-09-15T02:49:36.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497599+0000", 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497600+0000", 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6199999999999998e-07 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499791+0000", 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021917780000000001 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499856+0000", 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4332999999999995e-05 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499867+0000", 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1270999999999999e-05 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499962+0000", 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4900000000000003e-05 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499980+0000", 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8051999999999999e-05 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.518745+0000", 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018764533999999999 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.518769+0000", 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4868999999999999e-05 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 83/79 ECSubWrite(tid=486, reqid=client.4526.0:823, at_version=83'105, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.548783+0000", 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.0686061850000002, 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010846891000000001, 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.691 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548783+0000", 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548783+0000", 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548785+0000", 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8870000000000001e-06 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549317+0000", 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053127600000000001 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549412+0000", 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5493000000000002e-05 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549431+0000", 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9167000000000001e-05 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549548+0000", 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011657799999999999 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549574+0000", 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6165000000000001e-05 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.559591+0000", 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010017163000000001 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.559630+0000", 2024-09-15T02:49:36.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9162000000000002e-05 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.18s2 87/76 ECSubWrite(tid=27, reqid=client.4526.0:867, at_version=87'60, trim_to=0'0, roll_forward_to=85'56))", 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:31.306351+0000", 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3110382600000001, 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017165407000000001, 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.306351+0000", 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.306351+0000", 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.306352+0000", 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.111e-06 2024-09-15T02:49:36.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.307230+0000", 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00087759000000000003 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.307271+0000", 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0771e-05 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.307278+0000", 2024-09-15T02:49:36.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4510000000000003e-06 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.307344+0000", 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.546e-05 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.307355+0000", 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1223000000000001e-05 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.323484+0000", 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016128725 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.323517+0000", 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3076e-05 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s3 87/79 ECSubWrite(tid=518, reqid=client.4526.0:855, at_version=87'116, trim_to=0'0, roll_forward_to=85'110))", 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:31.312110+0000", 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3052795709999998, 2024-09-15T02:49:36.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010872956, 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.312110+0000", 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.312110+0000", 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.312111+0000", 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1209999999999999e-06 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.312178+0000", 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6382000000000002e-05 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.312231+0000", 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3035999999999998e-05 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.312239+0000", 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2830000000000001e-06 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313192+0000", 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00095308199999999995 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313310+0000", 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000117593 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.322969+0000", 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096590679999999998 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.322983+0000", 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4391e-05 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.18s2 87/76 ECSubWrite(tid=28, reqid=client.4526.0:868, at_version=87'61, trim_to=0'0, roll_forward_to=85'56))", 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:31.313272+0000", 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3041178899999997, 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010732495999999999, 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313272+0000", 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313272+0000", 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313273+0000", 2024-09-15T02:49:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0699999999999996e-07 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313333+0000", 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0752999999999999e-05 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313397+0000", 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4185999999999998e-05 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.313405+0000", 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7840000000000004e-06 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.319550+0000", 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061443540000000003 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.319608+0000", 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8777999999999999e-05 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.323984+0000", 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043758360000000001 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.324004+0000", 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9998000000000001e-05 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs1 87/76 ECSubWrite(tid=523, reqid=client.4526.0:880, at_version=87'106, trim_to=0'0, roll_forward_to=85'104))", 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:31.338362+0000", 2024-09-15T02:49:36.709 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2790276010000001, 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029316723999999999, 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.338362+0000", 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.338362+0000", 2024-09-15T02:49:36.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.338364+0000", 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6020000000000001e-06 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.339038+0000", 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067448200000000001 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.339096+0000", 2024-09-15T02:49:36.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8184999999999999e-05 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.339107+0000", 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0356999999999999e-05 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.339172+0000", 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5332000000000004e-05 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.339182+0000", 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0491e-05 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.367653+0000", 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028470966 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:31.367679+0000", 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5309000000000001e-05 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.713 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:36.714 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:36.714 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:36.715 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:36.715 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:36.715 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:36.716 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:36.716 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:36.707+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:36.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:36.816 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:36.812+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:36.823 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:36.823 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:36.823 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:36.823 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:36.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:36.864 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:36.895 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:36.918 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:36.916+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:36.921 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:36.918+0000 7fd7c2dd1540 -1 Falling back to public interface 2024-09-15T02:49:36.925 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:36.928 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:36.928 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:36.928 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:36.928 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:36.928 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:36.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:36.955 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:36.955+0000 7fabb7fff700 0 -- 172.21.15.6:0/3002462830 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fabb8193350 msgr2=0x7fabb8195750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:37.019 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:37.014+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:49:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 52.453318522000004, 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014940116999999999, 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577945+0000", 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.037e-06 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035186899999999999 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578459+0000", 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016206200000000001 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578482+0000", 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3070000000000001e-05 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578643+0000", 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016161999999999999 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578670+0000", 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592839+0000", 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014168557 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592884+0000", 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4786000000000001e-05 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.397056386999999, 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016865508000000001, 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0999999999999999e-07 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634501+0000", 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029489499999999999 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634605+0000", 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103488 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634624+0000", 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9530000000000001e-05 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634715+0000", 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1284000000000002e-05 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634741+0000", 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5350999999999998e-05 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651036+0000", 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016295245999999999 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651071+0000", 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5203999999999997e-05 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=180, reqid=client.4526.0:385, at_version=49'53, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.271661055000003, 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010517344, 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759603+0000", 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7179999999999999e-06 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759826+0000", 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0002231 2024-09-15T02:49:37.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759971+0000", 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014553100000000001 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760158+0000", 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018628199999999999 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760305+0000", 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014718500000000001 2024-09-15T02:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760353+0000", 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8418999999999998e-05 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770084+0000", 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097306449999999996 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770118+0000", 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4464000000000001e-05 2024-09-15T02:49:37.046 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.746822057999999, 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015418308, 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284441+0000", 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.251e-06 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284558+0000", 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011635 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284645+0000", 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6879000000000006e-05 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284653+0000", 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8100000000000004e-06 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284760+0000", 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106585 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284774+0000", 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4260000000000001e-05 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299827+0000", 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015052345 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299858+0000", 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1828000000000001e-05 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=62, reqid=client.4526.0:418, at_version=50'6, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.746699456000002, 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015317982000000001, 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.051 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:37.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284595+0000", 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2381e-05 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284673+0000", 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7855000000000006e-05 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284681+0000", 2024-09-15T02:49:37.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9910000000000002e-06 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289497+0000", 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048157060000000003 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289509+0000", 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.166e-05 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299868+0000", 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010359454000000001 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299881+0000", 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2622e-05 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=63, reqid=client.4526.0:419, at_version=50'7, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:37.055 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.741749972999997, 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014955008000000001, 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:37.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289513+0000", 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.569999999999999e-07 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289540+0000", 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6984000000000001e-05 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289594+0000", 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4117999999999999e-05 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289608+0000", 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4219999999999999e-05 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299620+0000", 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010011978 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299633+0000", 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2564e-05 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304436+0000", 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048034269999999999 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304467+0000", 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0759999999999997e-05 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.740299077000003, 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014581759999999999, 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290964+0000", 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6499999999999998e-07 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291188+0000", 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224513 2024-09-15T02:49:37.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291233+0000", 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4665e-05 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291241+0000", 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8019999999999992e-06 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291323+0000", 2024-09-15T02:49:37.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2284e-05 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291336+0000", 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2906e-05 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305516+0000", 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014179916000000001 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305545+0000", 2024-09-15T02:49:37.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8809000000000001e-05 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.739910977000001, 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014221952, 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291352+0000", 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6000000000000004e-07 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291563+0000", 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000211387 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291616+0000", 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3294000000000002e-05 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291625+0000", 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8559999999999996e-06 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299094+0000", 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007469084 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299105+0000", 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0226e-05 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305552+0000", 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064471270000000004 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305573+0000", 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1418000000000001e-05 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.616183491999998, 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024987760000000001, 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.068 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:37.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7300000000000001e-07 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420805+0000", 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057259099999999999 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420899+0000", 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4377999999999997e-05 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420915+0000", 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5883e-05 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.421162+0000", 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024698199999999997 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.421192+0000", 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9342000000000001e-05 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440023+0000", 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018831260999999998 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440066+0000", 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3531e-05 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/47 ECSubWrite(tid=288, reqid=client.4526.0:542, at_version=60'32, trim_to=0'0, roll_forward_to=48'29))", 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:37.072 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.606663408999999, 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011886391, 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424601+0000", 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1330000000000002e-06 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424662+0000", 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1493000000000002e-05 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424768+0000", 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010543 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424786+0000", 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.876e-05 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424886+0000", 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9922999999999999e-05 2024-09-15T02:49:37.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424906+0000", 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9110000000000002e-05 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436423+0000", 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011517135 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436485+0000", 2024-09-15T02:49:37.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2407000000000001e-05 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=54, reqid=client.4526.0:537, at_version=60'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.605760963000002, 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014882912, 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425503+0000", 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425746+0000", 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024368799999999999 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425834+0000", 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7188999999999995e-05 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425848+0000", 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4107000000000001e-05 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429379+0000", 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035311700000000001 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429394+0000", 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5582000000000001e-05 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440351+0000", 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010956702 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440384+0000", 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2873999999999997e-05 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/47 ECSubWrite(tid=289, reqid=client.4526.0:543, at_version=60'33, trim_to=0'0, roll_forward_to=48'29))", 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.602737197000003, 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011573594, 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.081 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428526+0000", 2024-09-15T02:49:37.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.482e-06 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428559+0000", 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2526e-05 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428636+0000", 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.674e-05 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428655+0000", 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9664999999999999e-05 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429356+0000", 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070097400000000004 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429370+0000", 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3755e-05 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440078+0000", 2024-09-15T02:49:37.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010708255999999999 2024-09-15T02:49:37.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440098+0000", 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0196000000000001e-05 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.15s1 77/76 [PushOp(3:a906e5df:::smithi19125844-36:c, version: 72'52, data_included: [0~147456], data_size: 147456, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:a906e5df:::smithi19125844-36:c@72'52, size: 589824, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:589824, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.089 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.394602+0000", 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.636660375, 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010446046, 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394602+0000", 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394602+0000", 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394603+0000", 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6700000000000001e-06 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394838+0000", 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023451799999999999 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394925+0000", 2024-09-15T02:49:37.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394948+0000", 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3206000000000001e-05 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.395097+0000", 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014951999999999999 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.405048+0000", 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099504439999999993 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.15s1 77/76 [PushOp(3:a906e5df:::smithi19125844-36:head, version: 72'53, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a906e5df:::smithi19125844-36:head@72'53, size: 1835008, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.411603+0000", 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.619659113999999, 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013109686000000001, 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.093 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411603+0000", 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411603+0000", 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411604+0000", 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-15T02:49:37.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411872+0000", 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000268299 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411956+0000", 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4110999999999995e-05 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411979+0000", 2024-09-15T02:49:37.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2889999999999999e-05 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.412117+0000", 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013782399999999999 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.424713+0000", 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012595653 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s2 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 23, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.568702+0000", 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.462559947999999, 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013310231, 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.568702+0000", 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.568702+0000", 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.568704+0000", 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569645+0000", 2024-09-15T02:49:37.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094167199999999995 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569735+0000", 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9248999999999999e-05 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569758+0000", 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569898+0000", 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001393 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.582012+0000", 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012114860999999999 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.100 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s2 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.773456+0000", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.25780585, 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013906271, 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773456+0000", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773456+0000", 2024-09-15T02:49:37.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773458+0000", 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8500000000000001e-06 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774079+0000", 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062117800000000001 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774123+0000", 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3902000000000001e-05 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774136+0000", 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3055e-05 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774351+0000", 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021454300000000001 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.787363+0000", 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013011742999999999 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 83/79 ECSubWrite(tid=486, reqid=client.4526.0:823, at_version=83'105, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.548804+0000", 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.482457643, 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095958910000000005, 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.104 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548804+0000", 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548804+0000", 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548806+0000", 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.646e-06 2024-09-15T02:49:37.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549034+0000", 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022757300000000001 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549110+0000", 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6501999999999994e-05 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549123+0000", 2024-09-15T02:49:37.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2765999999999999e-05 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549221+0000", 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7750000000000004e-05 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549244+0000", 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2900000000000001e-05 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.558366+0000", 2024-09-15T02:49:37.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091223180000000008 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.558400+0000", 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4436e-05 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 83/79 ECSubWrite(tid=487, reqid=client.4526.0:824, at_version=83'106, trim_to=0'0, roll_forward_to=83'104))", 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.560882+0000", 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.4703797929999993, 2024-09-15T02:49:37.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099151139999999992, 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.560882+0000", 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.560882+0000", 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.560884+0000", 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0090000000000002e-06 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561033+0000", 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014894800000000001 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561189+0000", 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015607699999999999 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561232+0000", 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2800999999999999e-05 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561348+0000", 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115623 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561356+0000", 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2679999999999994e-06 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570768+0000", 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094117799999999998 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570797+0000", 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9607999999999998e-05 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.112 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 83/76 ECSubWrite(tid=130, reqid=client.4526.0:837, at_version=83'97, trim_to=0'0, roll_forward_to=83'95))", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.623549+0000", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.4077127600000008, 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012980457000000001, 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.623549+0000", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.623549+0000", 2024-09-15T02:49:37.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.623550+0000", 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1400000000000001e-06 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624037+0000", 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048631199999999997 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624120+0000", 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3255000000000002e-05 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624154+0000", 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3921999999999998e-05 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624234+0000", 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0131999999999998e-05 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624247+0000", 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2692e-05 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.636483+0000", 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012236629000000001 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.636530+0000", 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6375000000000002e-05 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.116 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.15s1 88/76 ECSubWrite(tid=175, reqid=client.4526.0:903, at_version=88'61, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.302753+0000", 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7285088779999995, 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095794560000000001, 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.302753+0000", 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.302753+0000", 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.302754+0000", 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.28e-06 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303118+0000", 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036316400000000002 2024-09-15T02:49:37.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303181+0000", 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3756000000000004e-05 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303192+0000", 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0783e-05 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303258+0000", 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5732e-05 2024-09-15T02:49:37.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303266+0000", 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6369999999999999e-06 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.312286+0000", 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090208379999999998 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.312333+0000", 2024-09-15T02:49:37.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6266000000000001e-05 2024-09-15T02:49:37.121 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.121 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.121 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.121 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.121 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.121 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:37.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:37.123 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:37.115+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:37.223 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:37.216+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:37.230 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.230 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:37.230 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:37.230 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:37.240 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:373: done (1 left) 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:374: read oid 41 snap -1 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:374: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:375: setattr oid 23 current snap is 37 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:371: finishing write tid 1 to smithi19125844-14 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:371: finishing write tid 2 to smithi19125844-14 2024-09-15T02:49:37.247 INFO:tasks.rados.rados.0.smithi191.stdout:371: finishing write tid 3 to smithi19125844-14 2024-09-15T02:49:37.248 INFO:tasks.rados.rados.0.smithi191.stdout:371: finishing write tid 4 to smithi19125844-14 2024-09-15T02:49:37.248 INFO:tasks.rados.rados.0.smithi191.stdout:371: finishing write tid 5 to smithi19125844-14 2024-09-15T02:49:37.248 INFO:tasks.rados.rados.0.smithi191.stdout:371: finishing write tid 6 to smithi19125844-14 2024-09-15T02:49:37.248 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 14 v 61 (ObjNum 135 snap 37 seq_num 135) dirty exists 2024-09-15T02:49:37.248 INFO:tasks.rados.rados.0.smithi191.stdout:371: left oid 14 (ObjNum 135 snap 37 seq_num 135) 2024-09-15T02:49:37.252 INFO:tasks.rados.rados.0.smithi191.stdout:371: done (2 left) 2024-09-15T02:49:37.252 INFO:tasks.rados.rados.0.smithi191.stdout:376: copy_from oid 50 from oid 36 current snap is 37 2024-09-15T02:49:37.252 INFO:tasks.rados.rados.0.smithi191.stdout:377: rmattr oid 47 current snap is 37 2024-09-15T02:49:37.252 INFO:tasks.rados.rados.0.smithi191.stdout:377: done (3 left) 2024-09-15T02:49:37.252 INFO:tasks.rados.rados.0.smithi191.stdout:378: snap_create 2024-09-15T02:49:37.257 INFO:tasks.rados.rados.0.smithi191.stdout:374: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T02:49:37.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:49:37.262 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 23 v 114 (ObjNum 118 snap 26 seq_num 118) dirty exists 2024-09-15T02:49:37.324 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:37.320+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:37.337 INFO:tasks.rados.rados.0.smithi191.stdout:376: finishing copy_from racing read to smithi19125844-50 2024-09-15T02:49:37.339 INFO:tasks.rados.rados.0.smithi191.stdout:376: finishing copy_from to smithi19125844-50 2024-09-15T02:49:37.339 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 50 v 73 (ObjNum 130 snap 33 seq_num 130) dirty exists 2024-09-15T02:49:37.353 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.353 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:37.353 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:37.353 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:37.353 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:37.362 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:37.425 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:37.419+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:37.514 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=20, reqid=client.4526.0:249, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.919153227000002, 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020019123999999999, 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:37.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592951+0000", 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054339899999999997 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593028+0000", 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6892999999999997e-05 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593044+0000", 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6407999999999999e-05 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595661+0000", 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002617032 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595686+0000", 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5358999999999999e-05 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612387+0000", 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016700194000000002 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612426+0000", 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9138999999999999e-05 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.519 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=21, reqid=client.4526.0:250, at_version=41'14, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.907240475999998, 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086779219999999994, 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604321+0000", 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.902e-06 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604517+0000", 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019603099999999999 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604572+0000", 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4144000000000002e-05 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604581+0000", 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4360000000000008e-06 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608062+0000", 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034810710000000001 2024-09-15T02:49:37.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608079+0000", 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7059999999999999e-05 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612964+0000", 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004885197 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612997+0000", 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3080999999999997e-05 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.911593801000002, 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080113409999999999, 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599967+0000", 2024-09-15T02:49:37.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3120000000000001e-06 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600013+0000", 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126999999999997e-05 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600081+0000", 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8295999999999998e-05 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600094+0000", 2024-09-15T02:49:37.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3522e-05 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600358+0000", 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026319700000000001 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600388+0000", 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029e-05 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607948+0000", 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075605949999999998 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607977+0000", 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9263000000000001e-05 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:37.528 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.240669165, 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012340179999999999, 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:37.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0999999999999999e-07 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271155+0000", 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026426800000000001 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271179+0000", 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3893999999999999e-05 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271190+0000", 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0525e-05 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271256+0000", 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6328999999999995e-05 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271268+0000", 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1831000000000001e-05 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283197+0000", 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011929294999999999 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283231+0000", 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3627999999999997e-05 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.532 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.18s1 77/76 ECSubWrite(tid=10, reqid=client.4526.0:769, at_version=77'48, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.022406350000001, 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017491599, 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:37.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489188+0000", 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3583e-05 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489252+0000", 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3886000000000005e-05 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489261+0000", 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6680000000000007e-06 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.492869+0000", 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036079990000000002 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.492877+0000", 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5699999999999993e-06 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506625+0000", 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013747786 2024-09-15T02:49:37.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506645+0000", 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0304000000000001e-05 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.18s1 77/76 [PushOp(3:1c51ca92:::smithi19125844-38:26, version: 74'43, data_included: [0~106496], data_size: 106496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:1c51ca92:::smithi19125844-38:26@74'43, size: 425984, copy_subset: [], clone_subset: {}, snapset: 27=[]:{26=[26,25,24,21,1d,18,b],27=[27]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:425984, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.491500+0000", 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.020059742000001, 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015075897, 2024-09-15T02:49:37.537 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491500+0000", 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491500+0000", 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491501+0000", 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6000000000000002e-07 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491651+0000", 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015037699999999999 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491694+0000", 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2834999999999999e-05 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491704+0000", 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3740000000000004e-06 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493618+0000", 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019141970000000001 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506576+0000", 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012958254000000001 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.540 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.18s1 77/76 [PushOp(3:1a9d0ab8:::smithi19125844-25:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:1a9d0ab8:::smithi19125844-25:head@72'42, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.493005+0000", 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.018555343999999, 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013545830999999999, 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493005+0000", 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493005+0000", 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493006+0000", 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.017e-06 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493359+0000", 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035336500000000002 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493406+0000", 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7222999999999998e-05 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493418+0000", 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.163e-05 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.496752+0000", 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033343029999999998 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506550+0000", 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097982929999999996 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.544 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.18s1 77/76 ECSubWrite(tid=15, reqid=client.4526.0:770, at_version=77'49, trim_to=0'0, roll_forward_to=77'48))", 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.514210+0000", 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.997349664, 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057267200000000002, 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.514210+0000", 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.514210+0000", 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.514211+0000", 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1340000000000001e-06 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.514254+0000", 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2517e-05 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.514312+0000", 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7757000000000003e-05 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.514322+0000", 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0246e-05 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.518702+0000", 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043804969999999997 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.518718+0000", 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5492e-05 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.519907+0000", 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011886380000000001 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.519937+0000", 2024-09-15T02:49:37.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0439000000000001e-05 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.013719106, 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019140009999999999, 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6e-07 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499086+0000", 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012450510000000001 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499145+0000", 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8804999999999997e-05 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499154+0000", 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9239999999999996e-06 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499307+0000", 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015314899999999999 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499329+0000", 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2248000000000001e-05 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.516950+0000", 2024-09-15T02:49:37.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017620523999999999 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.516981+0000", 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1149000000000003e-05 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.12s3 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.522939+0000", 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.98862059, 2024-09-15T02:49:37.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058133600000000001, 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.522939+0000", 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.522939+0000", 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.522940+0000", 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1260000000000001e-06 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523514+0000", 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057330300000000005 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523569+0000", 2024-09-15T02:49:37.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5475e-05 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523581+0000", 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2225999999999999e-05 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523632+0000", 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0643000000000003e-05 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523644+0000", 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2092999999999999e-05 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.528730+0000", 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050854580000000002 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.528753+0000", 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3036000000000001e-05 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.557 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.12s3 78/77 ECSubWrite(tid=130, reqid=client.4526.0:744, at_version=78'42, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.534465+0000", 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.977094781, 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071005670000000003, 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534465+0000", 2024-09-15T02:49:37.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534465+0000", 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534466+0000", 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534989+0000", 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052318000000000004 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535051+0000", 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2113999999999995e-05 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535073+0000", 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1359e-05 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535149+0000", 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5894000000000002e-05 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535168+0000", 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8838000000000002e-05 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.541535+0000", 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063670369999999999 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.541566+0000", 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1164000000000002e-05 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.12s3 78/77 ECSubRead(tid=134, to_read={3:48b768c1:::smithi19125844-48:head=0,4096,0}, subchunks={3:48b768c1:::smithi19125844-48:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.561746+0000", 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.949814014999999, 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090836619999999993, 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.562 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561746+0000", 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561746+0000", 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561747+0000", 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561757+0000", 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0322999999999999e-05 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570342+0000", 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085848009999999995 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570355+0000", 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3115000000000001e-05 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570469+0000", 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113448 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570830+0000", 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036096199999999999 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.565 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 23, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.569080+0000", 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.942480153, 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19609521199999999, 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569080+0000", 2024-09-15T02:49:37.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569080+0000", 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569081+0000", 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.46e-06 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570304+0000", 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001222919 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.748181+0000", 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.17787734099999999 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.748210+0000", 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8915999999999999e-05 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.748415+0000", 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020410200000000001 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.765175+0000", 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016760474000000001 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.569 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.773808+0000", 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.737752064, 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097636119999999996, 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773808+0000", 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773808+0000", 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773810+0000", 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8169999999999999e-06 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774581+0000", 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077112000000000003 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774666+0000", 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5634e-05 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774690+0000", 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3315999999999998e-05 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774786+0000", 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6129000000000001e-05 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.783571+0000", 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087855959999999997 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.573 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [524288~172032], data_size: 172032, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2785280, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.798905+0000", 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.712654643, 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092489289999999995, 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.798905+0000", 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.798905+0000", 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.798907+0000", 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799185+0000", 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027742700000000001 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799269+0000", 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4704999999999996e-05 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799288+0000", 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8389e-05 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799387+0000", 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9470999999999997e-05 2024-09-15T02:49:37.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.808154+0000", 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087671139999999995 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 43, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.828736+0000", 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.682824265000001, 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016867099999999999, 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.577 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.828736+0000", 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.828736+0000", 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.828737+0000", 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7770000000000001e-06 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.829950+0000", 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012128830000000001 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.830039+0000", 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8721000000000007e-05 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.830063+0000", 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4193e-05 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.830199+0000", 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013589 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.845603+0000", 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015403636 2024-09-15T02:49:37.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.850192+0000", 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.661368056000001, 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006788924, 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850192+0000", 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850192+0000", 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850193+0000", 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6700000000000002e-07 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850699+0000", 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050657600000000001 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850740+0000", 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0948000000000002e-05 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850749+0000", 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4370000000000007e-06 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850805+0000", 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6122000000000002e-05 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.856981+0000", 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061758739999999996 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.584 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.861481+0000", 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.650079138000001, 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074314619999999998, 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861481+0000", 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861481+0000", 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861482+0000", 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.083e-06 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861799+0000", 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000317368 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861844+0000", 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5117000000000002e-05 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861853+0000", 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861899+0000", 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6385999999999999e-05 2024-09-15T02:49:37.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.868912+0000", 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070131029999999997 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [786432~204800], data_size: 204800, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3964928, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.873803+0000", 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.637756736, 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062309849999999996, 2024-09-15T02:49:37.588 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.873803+0000", 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.873803+0000", 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.873804+0000", 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.08e-06 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874311+0000", 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050656599999999996 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874353+0000", 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2154999999999999e-05 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874361+0000", 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2779999999999993e-06 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874410+0000", 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8786999999999999e-05 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.880034+0000", 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056241190000000003 2024-09-15T02:49:37.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 93/76 ECSubWrite(tid=156, reqid=client.4526.0:952, at_version=93'25, trim_to=0'0, roll_forward_to=85'23))", 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.225843987, 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038285843999999999, 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4900000000000001e-07 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289183+0000", 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034662299999999998 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289249+0000", 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6700999999999997e-05 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289261+0000", 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1240000000000001e-05 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289437+0000", 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017670500000000001 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289466+0000", 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8433999999999999e-05 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.323978+0000", 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034512481999999997 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.324002+0000", 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3603000000000001e-05 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.596 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:37.598 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:37.596+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:37.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:37.699 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:37.694+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:37.705 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.705 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:37.705 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:37.705 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:37.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:37.800 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:37.796+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:37.814 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.814 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:37.814 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:37.814 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:37.814 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:37.823 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:37.901 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:37.896+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:37.912 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:37.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:388 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.168296838000003, 2024-09-15T02:49:37.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.205927366, 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 388 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:37.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752805+0000", 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8480000000000003e-06 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752820+0000", 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752827+0000", 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8059999999999997e-06 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770449+0000", 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017621527000000001 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770567+0000", 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118576 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776872+0000", 2024-09-15T02:49:37.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063044340000000003 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776986+0000", 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011429699999999999 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958728+0000", 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.181741654 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:389 3.7s0 3:e10d9d4d:::smithi19125844-8:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.168261577000003, 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47945425899999999, 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 389 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752836+0000", 2024-09-15T02:49:37.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752842+0000", 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8579999999999996e-06 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752860+0000", 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7816e-05 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752867+0000", 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0260000000000002e-06 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770793+0000", 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017925963999999999 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770931+0000", 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137978 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955584+0000", 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.184652765 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.959571+0000", 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003987488 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232070+0000", 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.272499243 2024-09-15T02:49:37.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232290+0000", 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021937500000000001 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:390 3.7s0 3:e10d9d4d:::smithi19125844-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.168223386999998, 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20558775300000001, 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.935 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 390 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8199999999999999e-07 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752878+0000", 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4489999999999999e-06 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752888+0000", 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0122000000000001e-05 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752894+0000", 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9220000000000002e-06 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771143+0000", 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018248665000000001 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771198+0000", 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5158000000000003e-05 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958258+0000", 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18706018699999999 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958359+0000", 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101025 2024-09-15T02:49:37.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958461+0000", 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102543 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:402 3.1s0 3:801a1da7:::smithi19125844-39:e [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.162845623999999, 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52960550699999998, 2024-09-15T02:49:37.940 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 402 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758252+0000", 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5000000000000002e-07 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758257+0000", 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3800000000000004e-06 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758267+0000", 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0455000000000001e-05 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758604+0000", 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033705300000000003 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761065+0000", 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024613840000000001 2024-09-15T02:49:37.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761208+0000", 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014282499999999999 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287857+0000", 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52664865999999999 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 49/33 ECSubReadReply(tid=183, attrs_read=0))", 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.157873123999998, 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52471420199999996, 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763572+0000", 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034791800000000001 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286378+0000", 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52280558799999999 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286384+0000", 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286558+0000", 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017370499999999999 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287938+0000", 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013805550000000001 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:433 3.7s0 3:ec23d144:::smithi19125844-41:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.638103751000003, 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27191602199999998, 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.948 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282994+0000", 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1790000000000001e-06 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283000+0000", 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0479999999999998e-06 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283008+0000", 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1440000000000003e-06 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283012+0000", 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7129999999999996e-06 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298201+0000", 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015188495 2024-09-15T02:49:37.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298355+0000", 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015441200000000001 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298951+0000", 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059618300000000002 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300677+0000", 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001725285 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554804+0000", 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25412710900000002 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554909+0000", 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105454 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.953 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:434 3.7s0 3:ec23d144:::smithi19125844-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.638079828000002, 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27651220700000001, 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 434 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.954 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283018+0000", 2024-09-15T02:49:37.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0500000000000003e-07 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283021+0000", 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.959e-06 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283028+0000", 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0169999999999999e-06 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283032+0000", 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0509999999999998e-06 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299373+0000", 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016340659 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299481+0000", 2024-09-15T02:49:37.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108218 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554991+0000", 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25551037700000001 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.555256+0000", 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026495400000000002 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559529+0000", 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042733670000000001 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:437 3.6s0 3:6e9f7d10:::smithi19125844-33:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.638012891000002, 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20344801100000001, 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.959 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 437 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:37.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283085+0000", 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283088+0000", 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.993e-06 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283094+0000", 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5810000000000002e-06 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6289999999999999e-06 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284035+0000", 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093763400000000002 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284100+0000", 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4882999999999995e-05 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486532+0000", 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20243185599999999 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:438 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [delete] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e50)", 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:37.963 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.637979082000001, 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27354936000000002, 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 438 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8100000000000003e-07 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283121+0000", 2024-09-15T02:49:37.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8830000000000002e-06 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283129+0000", 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3390000000000004e-06 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283133+0000", 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299523+0000", 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389989000000001 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299625+0000", 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010208 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300116+0000", 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049149000000000003 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300972+0000", 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085554200000000004 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556582+0000", 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25561025399999998 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556667+0000", 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5208000000000001e-05 2024-09-15T02:49:37.968 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.6s0 50/33 ECSubReadReply(tid=200, attrs_read=0))", 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.625809488999998, 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19127902199999999, 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2600000000000001e-07 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295338+0000", 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9951999999999999e-05 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485520+0000", 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19018270600000001 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485530+0000", 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0820000000000005e-06 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485655+0000", 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012580999999999999 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486567+0000", 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000911246 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.972 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=203, last_complete=50'62, committed=1, applied=1))", 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.617764307000002, 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25159346799999999, 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:37.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303336+0000", 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9550000000000001e-06 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554637+0000", 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25130088299999997 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554647+0000", 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9329999999999996e-06 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554704+0000", 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6514000000000001e-05 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554926+0000", 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222647 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 50/33 ECSubWriteReply(tid=205, last_complete=50'35, committed=1, applied=1))", 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.601756877, 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.23186345, 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319341+0000", 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6700000000000003e-07 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319347+0000", 2024-09-15T02:49:37.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6879999999999996e-06 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550925+0000", 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.231578591 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550946+0000", 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0903e-05 2024-09-15T02:49:37.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551110+0000", 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016408300000000001 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551204+0000", 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3418000000000006e-05 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=208, last_complete=50'64, committed=1, applied=1))", 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.361899711, 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.60425413100000003, 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559198+0000", 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9500000000000001e-07 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559203+0000", 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.13e-06 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163222+0000", 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.604018638 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163237+0000", 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4622e-05 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163384+0000", 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014714099999999999 2024-09-15T02:49:37.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163451+0000", 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7805000000000006e-05 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 54/53 ECSubWrite(tid=87, reqid=client.4526.0:461, at_version=0'0, trim_to=0'0, roll_forward_to=54'39))", 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.999997043999997, 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.41018363000000002, 2024-09-15T02:49:37.984 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921102+0000", 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921127+0000", 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4987e-05 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329820+0000", 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.40869367400000001 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329833+0000", 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2669e-05 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329958+0000", 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012490099999999999 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329981+0000", 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2463000000000002e-05 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331170+0000", 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001189701 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331284+0000", 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113435 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.988 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPushReply(3.1s0 54/53 [PushReplyOp(3:801a1da7:::smithi19125844-39:head)])", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.626280260999998, 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99244340499999995, 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:37.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294818+0000", 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294837+0000", 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9052999999999999e-05 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286615+0000", 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99177708899999995 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286629+0000", 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4583000000000001e-05 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286721+0000", 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1640999999999999e-05 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.287260+0000", 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053939199999999997 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 55/53 ECSubRead(tid=88, to_read={3:94f52c52:::smithi19125844-20:head=0,110592,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.591209788, 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59246917700000001, 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.992 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329888+0000", 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329900+0000", 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.188e-05 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921816+0000", 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59191607099999999 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921831+0000", 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4765e-05 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922011+0000", 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018023900000000001 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922356+0000", 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034564200000000001 2024-09-15T02:49:37.995 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 62/53 ECSubWriteReply(tid=316, last_complete=62'23, committed=1, applied=1))", 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.563655291, 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58891104599999999, 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6400000000000002e-07 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357447+0000", 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9529999999999998e-06 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945262+0000", 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58781430300000004 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945281+0000", 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9366000000000001e-05 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945429+0000", 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014825600000000001 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.946353+0000", 2024-09-15T02:49:37.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092350400000000001 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.0.22:2 3.7s0 3:e10d9d4d:::smithi19125844-8:head [assert-version v25,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.445179601, 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.014934445, 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:38.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475920+0000", 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-09-15T02:49:38.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475928+0000", 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1750000000000005e-06 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475952+0000", 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4244e-05 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475960+0000", 2024-09-15T02:49:38.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0940000000000008e-06 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476051+0000", 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1032000000000005e-05 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476057+0000", 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3029999999999999e-06 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:38.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484706+0000", 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0086496460000001 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484954+0000", 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024785999999999997 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.490852+0000", 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058976979999999998 2024-09-15T02:49:38.004 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:912 3.bs0 3:d6271403:::smithi19125844-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.6056001340000003, 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18619464499999999, 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:38.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 912 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:38.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:38.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:38.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:38.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7899999999999998e-07 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315501+0000", 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2399999999999999e-06 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315519+0000", 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8094999999999999e-05 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315525+0000", 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8429999999999998e-06 2024-09-15T02:49:38.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.340958+0000", 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025432834000000001 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.341123+0000", 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016572000000000001 2024-09-15T02:49:38.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.499134+0000", 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.15801039 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.499395+0000", 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026174100000000001 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.501692+0000", 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022962030000000001 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 88/76 ECSubWriteReply(tid=531, last_complete=88'112, committed=1, applied=1))", 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4203513560000003, 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74370967399999999, 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:38.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:38.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7700000000000004e-07 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500750+0000", 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8229999999999997e-06 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244204+0000", 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74345404000000004 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244216+0000", 2024-09-15T02:49:38.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1423000000000001e-05 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244348+0000", 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000132464 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244455+0000", 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107347 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:38.015 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:38.016 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:38.017 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:38.018 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:38.012+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:38.118 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:38.114+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:38.126 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:38.126 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:38.136 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:38.219 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:38.216+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:38.233 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:38.233 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:38.233 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:38.233 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:38.233 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:38.242 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:38.255 INFO:tasks.rados.rados.0.smithi191.stdout:374: done (3 left) 2024-09-15T02:49:38.255 INFO:tasks.rados.rados.0.smithi191.stdout:375: done (2 left) 2024-09-15T02:49:38.256 INFO:tasks.rados.rados.0.smithi191.stdout:376: done (1 left) 2024-09-15T02:49:38.256 INFO:tasks.rados.rados.0.smithi191.stdout:378: done (0 left) 2024-09-15T02:49:38.256 INFO:tasks.rados.rados.0.smithi191.stdout:379: rollback oid 24 current snap is 38 2024-09-15T02:49:38.256 INFO:tasks.rados.rados.0.smithi191.stdout:rollback oid 24 to 21 2024-09-15T02:49:38.256 INFO:tasks.rados.rados.0.smithi191.stdout:380: snap_remove snap 23 2024-09-15T02:49:38.260 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:49:38.320 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:38.315+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:49:38.334 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 73.595529794000001, 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:38.335 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.336 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:49:38.337 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.338 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:403 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~606208] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.574791099999999, 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.195887642, 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.339 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 403 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:38.340 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757022+0000", 2024-09-15T02:49:38.341 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.593e-06 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757800+0000", 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077824999999999997 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757830+0000", 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0057e-05 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.342 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757844+0000", 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3947999999999999e-05 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757925+0000", 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0249999999999999e-05 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.758082+0000", 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015777599999999999 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.343 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759386+0000", 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001303217 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777348+0000", 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017962568000000002 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952681+0000", 2024-09-15T02:49:38.344 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17533264000000001 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952907+0000", 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000226343 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:404 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~720896] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.570572495999997, 2024-09-15T02:49:38.345 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.192160255, 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 404 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.346 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761280+0000", 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2138999999999998e-05 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762400+0000", 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001119451 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.347 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762433+0000", 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3238000000000003e-05 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762446+0000", 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3308e-05 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768118+0000", 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056721360000000004 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.348 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768285+0000", 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000166482 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769662+0000", 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013768140000000001 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780475+0000", 2024-09-15T02:49:38.349 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010813578000000001 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953275+0000", 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17280025199999999 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953398+0000", 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012285700000000001 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.350 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:405 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~49152] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.569341758, 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19129260000000001, 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 405 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.351 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762470+0000", 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4699999999999999e-06 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.352 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762527+0000", 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7151999999999998e-05 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762550+0000", 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2311999999999999e-05 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762559+0000", 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:38.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777614+0000", 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015055628999999999 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777696+0000", 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1840000000000002e-05 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778120+0000", 2024-09-15T02:49:38.354 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042372900000000002 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780527+0000", 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00240712 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953622+0000", 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17309514500000001 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.355 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953761+0000", 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013926300000000001 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:406 3.es0 3:71c1a91c:::smithi19125844-45:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.569241798, 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19225968700000001, 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.356 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 406 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762570+0000", 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.034e-06 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762575+0000", 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4079999999999997e-06 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762592+0000", 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.715e-05 2024-09-15T02:49:38.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762600+0000", 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6480000000000003e-06 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779660+0000", 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017060035000000001 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779757+0000", 2024-09-15T02:49:38.359 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6537000000000001e-05 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780137+0000", 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038070299999999998 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781414+0000", 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012764230000000001 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.360 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954649+0000", 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17323481900000001 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954828+0000", 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017992999999999999 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:407 3.es0 3:71c1a91c:::smithi19125844-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.569201925999998, 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19532122900000001, 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 407 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.362 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762610+0000", 2024-09-15T02:49:38.363 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762614+0000", 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0659999999999997e-06 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762626+0000", 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2490000000000001e-05 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.364 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762639+0000", 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2704000000000001e-05 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781806+0000", 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019166823999999999 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.782082+0000", 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027635000000000002 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955680+0000", 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17359843999999999 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955807+0000", 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012658299999999999 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.957930+0000", 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021228499999999999 2024-09-15T02:49:38.366 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.13.22:5 3.es0 3:740f3d5b:::smithi19125844-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.556670945999997, 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.177222558, 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.367 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775141+0000", 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.368 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775146+0000", 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5630000000000002e-06 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775159+0000", 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3509999999999999e-05 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775165+0000", 2024-09-15T02:49:38.369 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8130000000000001e-06 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778738+0000", 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003572867 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778948+0000", 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00021028299999999999 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952362+0000", 2024-09-15T02:49:38.370 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17341393199999999 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:529 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~802816] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 34.076985489000002, 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.030230086, 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.371 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 529 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254827+0000", 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6390000000000001e-06 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256167+0000", 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013402900000000001 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256196+0000", 2024-09-15T02:49:38.373 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8762999999999999e-05 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256203+0000", 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6229999999999997e-06 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036663860000000002 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.374 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.260012+0000", 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014185099999999999 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017728349999999999 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277478+0000", 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015693650999999999 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.375 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285001+0000", 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075225120000000003 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285055+0000", 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4536000000000001e-05 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:530 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~638976] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:38.376 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 34.073064334999998, 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026488489, 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 530 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:38.377 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258748+0000", 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.66e-06 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.378 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259495+0000", 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00074681400000000005 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259522+0000", 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7552999999999999e-05 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259538+0000", 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6220000000000001e-05 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.379 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269841+0000", 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010302477000000001 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269927+0000", 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5993000000000002e-05 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.270621+0000", 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00069455099999999996 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.380 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277769+0000", 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0071479769999999998 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285179+0000", 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074099320000000002 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285235+0000", 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5312000000000002e-05 2024-09-15T02:49:38.381 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:533 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 34.072105641, 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027459596999999999, 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.382 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 533 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.89e-07 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259707+0000", 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5460000000000001e-06 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259711+0000", 2024-09-15T02:49:38.384 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1439999999999996e-06 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259715+0000", 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3340000000000003e-06 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277889+0000", 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018173363000000001 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.385 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277948+0000", 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8913999999999999e-05 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286109+0000", 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0081616199999999996 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286156+0000", 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6928000000000002e-05 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.386 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.287165+0000", 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010083589999999999 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s2 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.797709511000001, 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027643175999999998, 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.387 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.388 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534111+0000", 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8090000000000004e-06 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534425+0000", 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031419599999999998 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542230+0000", 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0078045069999999996 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.389 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542239+0000", 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542375+0000", 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000135897 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542396+0000", 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1237999999999999e-05 2024-09-15T02:49:38.390 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561688+0000", 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019291689000000001 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561744+0000", 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6178999999999999e-05 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.391 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.8s2 77/76 ECSubRead(tid=258, to_read={3:173bfaf3:::smithi19125844-44:18=262144,262144,0}, subchunks={3:173bfaf3:::smithi19125844-44:18=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.755950753, 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119859681, 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.392 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575861+0000", 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0640000000000001e-06 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575876+0000", 2024-09-15T02:49:38.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5165e-05 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694960+0000", 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119084367 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694980+0000", 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9930000000000001e-05 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695069+0000", 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8737000000000001e-05 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695719+0000", 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065041800000000005 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=112, attrs_read=0))", 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.751460720000001, 2024-09-15T02:49:38.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13363014300000001, 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580351+0000", 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2419999999999999e-06 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580721+0000", 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00037001299999999998 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712609+0000", 2024-09-15T02:49:38.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13188792199999999 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712623+0000", 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.429e-05 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712782+0000", 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000158791 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.398 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.713980+0000", 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011978850000000001 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.584433475000001, 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21641586800000001, 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747379+0000", 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747806+0000", 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042687399999999999 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962958+0000", 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21515266799999999 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962974+0000", 2024-09-15T02:49:38.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6237999999999999e-05 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963059+0000", 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4968000000000004e-05 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963793+0000", 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00073351999999999998 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.12s5 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 15.80996843, 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028389666000000001, 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521843+0000", 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.122e-06 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522442+0000", 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00059853399999999996 2024-09-15T02:49:38.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533671+0000", 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011229075 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533686+0000", 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5265999999999999e-05 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533798+0000", 2024-09-15T02:49:38.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000112198 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533821+0000", 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2229999999999999e-05 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550165+0000", 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016344826 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550232+0000", 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6414999999999993e-05 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:836 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~573440] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.733174134, 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033215366000000003, 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 836 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:38.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598639+0000", 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1280000000000002e-06 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.409 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599731+0000", 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001092469 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599759+0000", 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8472000000000001e-05 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599774+0000", 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4612e-05 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608764+0000", 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089903399999999994 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608966+0000", 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020188699999999999 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.610666+0000", 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016996940000000001 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.630970+0000", 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020303799000000001 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631771+0000", 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00080119100000000003 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631852+0000", 2024-09-15T02:49:38.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0773999999999996e-05 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:837 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~688128] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.728258654999999, 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037719309999999999, 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.413 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 837 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603553+0000", 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604429+0000", 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000875577 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604454+0000", 2024-09-15T02:49:38.415 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4794999999999999e-05 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604464+0000", 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9199999999999999e-06 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621603+0000", 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017138823000000001 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621666+0000", 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2996000000000005e-05 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.622382+0000", 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071611400000000001 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631868+0000", 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0094864259999999992 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641117+0000", 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0092490960000000001 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641271+0000", 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015396299999999999 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.418 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:838 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~294912] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.726316467, 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.03629342, 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 838 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605496+0000", 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.415e-06 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605778+0000", 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028258100000000001 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605797+0000", 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8817999999999998e-05 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605805+0000", 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5229999999999995e-06 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626744+0000", 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020938365 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626872+0000", 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012857999999999999 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.627645+0000", 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077218100000000004 2024-09-15T02:49:38.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631967+0000", 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043226499999999999 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641643+0000", 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096753510000000004 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641788+0000", 2024-09-15T02:49:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014495600000000001 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:839 3.es0 3:7773fe95:::smithi19125844-37:head [setxattr _header (57) in=7b] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.725996387, 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.036686074999999999, 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.424 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 839 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605815+0000", 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1909999999999999e-06 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605822+0000", 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3369999999999996e-06 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605840+0000", 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7949e-05 2024-09-15T02:49:38.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605848+0000", 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6980000000000004e-06 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631010+0000", 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025161701000000002 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631081+0000", 2024-09-15T02:49:38.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0993000000000003e-05 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631379+0000", 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000298146 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.632106+0000", 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072663800000000005 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642331+0000", 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010225316 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642500+0000", 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000169106 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:840 3.es0 3:7773fe95:::smithi19125844-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:38.429 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 10.725952764000001, 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.039020892000000001, 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 840 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:38.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605859+0000", 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.02e-06 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605864+0000", 2024-09-15T02:49:38.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1000000000000003e-06 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605882+0000", 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8128999999999999e-05 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605891+0000", 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5550000000000003e-06 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631895+0000", 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026004328 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631958+0000", 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2603999999999999e-05 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643304+0000", 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01134697 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643450+0000", 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000145069 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.644879+0000", 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0014291169999999999 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.434 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.435 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.435 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.435 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.435 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:38.436 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:38.437 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:38.432+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:38.537 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:38.532+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:38.539 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:38.539 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:38.540 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:38.540 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:38.549 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:38.638 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:38.631+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:38.648 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:38.648 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:38.648 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:38.648 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:38.648 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:38.658 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:38.739 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:38.732+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:38.762 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:38.762 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:38.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:38.762 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:38.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.762 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:49:38.763 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:38.763 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.139665825999998, 2024-09-15T02:49:38.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:49:38.763 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.763 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.763 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:49:38.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:416 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~638976] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.497457337, 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026381341999999999, 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.767 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 416 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.768 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261066+0000", 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1159999999999999e-06 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262194+0000", 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001127783 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262245+0000", 2024-09-15T02:49:38.769 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0880000000000001e-05 2024-09-15T02:49:38.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.770 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.770 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262260+0000", 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5065999999999999e-05 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262382+0000", 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012151599999999999 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262403+0000", 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1344000000000001e-05 2024-09-15T02:49:38.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266706+0000", 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004303355 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266794+0000", 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7717000000000001e-05 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268260+0000", 2024-09-15T02:49:38.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001466406 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285160+0000", 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016899773 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287372+0000", 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022116140000000002 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.773 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287445+0000", 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2772000000000001e-05 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:417 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.488063386999997, 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028569959999999998, 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.774 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 417 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270459+0000", 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.035e-06 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271813+0000", 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001354037 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271834+0000", 2024-09-15T02:49:38.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0302e-05 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271844+0000", 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9070000000000002e-06 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274939+0000", 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003095169 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275041+0000", 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000102045 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276123+0000", 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010819880000000001 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285162+0000", 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009039432 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.778 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.298984+0000", 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013821599 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299027+0000", 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3445999999999997e-05 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.779 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:418 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~147456] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.486660886000003, 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027304571999999999, 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 418 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.780 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271861+0000", 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1829999999999999e-06 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.781 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272037+0000", 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017563799999999999 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272050+0000", 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3506000000000001e-05 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272057+0000", 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999998e-06 2024-09-15T02:49:38.782 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281725+0000", 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096680910000000002 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281887+0000", 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016169599999999999 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.783 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282772+0000", 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088466100000000002 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285439+0000", 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026669340000000001 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299113+0000", 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013674009000000001 2024-09-15T02:49:38.784 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299164+0000", 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1573999999999998e-05 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:419 3.0s0 3:038350ba:::smithi19125844-11:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.486455589000002, 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.031729234000000002, 2024-09-15T02:49:38.785 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 419 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.786 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272066+0000", 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9000000000000005e-07 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272070+0000", 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1740000000000002e-06 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.787 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272078+0000", 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5329999999999997e-06 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272083+0000", 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7859999999999996e-06 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284520+0000", 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012437190000000001 2024-09-15T02:49:38.788 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284611+0000", 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0816000000000006e-05 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284926+0000", 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031568500000000001 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287991+0000", 2024-09-15T02:49:38.789 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030649599999999998 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303704+0000", 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015713142999999999 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303794+0000", 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9957000000000006e-05 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.790 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:420 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.486432581999999, 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033404771999999999, 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 420 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.791 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272089+0000", 2024-09-15T02:49:38.792 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.51e-07 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272091+0000", 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1730000000000002e-06 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272096+0000", 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.904e-06 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.793 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272099+0000", 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2059999999999999e-06 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285286+0000", 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013187488000000001 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285374+0000", 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7392999999999995e-05 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.794 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304237+0000", 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018863347999999999 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304386+0000", 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014841 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.305493+0000", 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001107299 2024-09-15T02:49:38.795 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:429 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.486359235000002, 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022788739999999998, 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.796 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 429 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.797 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272162+0000", 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272166+0000", 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9269999999999998e-06 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.798 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272174+0000", 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6569999999999997e-06 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272178+0000", 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6260000000000003e-06 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274127+0000", 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001949295 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.799 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274178+0000", 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0547999999999998e-05 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.294950+0000", 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020772112999999998 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.800 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 50/33 ECSubReadReply(tid=60, attrs_read=0))", 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 44.481085966999999, 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017580023, 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.801 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0100000000000004e-07 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293753+0000", 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016317505999999999 2024-09-15T02:49:38.802 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293859+0000", 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105707 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293862+0000", 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4819999999999999e-06 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293910+0000", 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7763000000000003e-05 2024-09-15T02:49:38.803 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.295015+0000", 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011048640000000001 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.12.22:2 3.9s0 3:94f52c52:::smithi19125844-20:head [assert-version v18,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 39.430664315999998, 2024-09-15T02:49:38.804 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59485876100000001, 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.805 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327859+0000", 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.446e-06 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327869+0000", 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7629999999999995e-06 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327898+0000", 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9054e-05 2024-09-15T02:49:38.806 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327913+0000", 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4990999999999999e-05 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328005+0000", 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2015000000000006e-05 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328180+0000", 2024-09-15T02:49:38.807 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017567599999999999 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.922715+0000", 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59453481600000002 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:630 3.9s0 3:9c49cf3d:::smithi19125844-14:head [copy-from ver 31] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:38.808 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 26.364609754, 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021438329999999998, 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 630 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.809 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393915+0000", 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7570000000000002e-06 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393951+0000", 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6347999999999998e-05 2024-09-15T02:49:38.810 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393987+0000", 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6405999999999997e-05 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394003+0000", 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5956000000000001e-05 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.811 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394131+0000", 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012749300000000001 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394155+0000", 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3754999999999999e-05 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401719+0000", 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075645340000000004 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.812 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401977+0000", 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000257543 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409969+0000", 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007991873 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414563+0000", 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045945830000000002 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.813 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415199+0000", 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063606199999999996 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415349+0000", 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015002 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.814 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:631 3.9s0 3:9c49cf3d:::smithi19125844-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 26.364468815999999, 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017944699000000001, 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 631 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.815 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394053+0000", 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3179999999999999e-06 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.816 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394059+0000", 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9120000000000003e-06 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394075+0000", 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5809999999999999e-05 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394085+0000", 2024-09-15T02:49:38.817 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0586e-05 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394258+0000", 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017270999999999999 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394277+0000", 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9236999999999999e-05 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.818 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402333+0000", 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0080553299999999994 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402447+0000", 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00011408600000000001 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411771+0000", 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093238820000000004 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.819 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411860+0000", 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8857999999999995e-05 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411997+0000", 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013697000000000001 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 72/71 ECSubWriteReply(tid=134, last_complete=72'51, committed=1, applied=1))", 2024-09-15T02:49:38.820 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.465678783000001, 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060209891000000001, 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.821 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292843+0000", 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3099999999999996e-07 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292848+0000", 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6729999999999997e-06 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.822 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352898+0000", 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060049335000000002 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352910+0000", 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.256e-05 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352988+0000", 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7831999999999996e-05 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.823 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.353052+0000", 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3559999999999995e-05 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:688 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 20.31964777, 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037318727000000003, 2024-09-15T02:49:38.824 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 688 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.825 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438875+0000", 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5809999999999999e-06 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438889+0000", 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3444999999999999e-05 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.826 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438927+0000", 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7842000000000001e-05 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438940+0000", 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3473e-05 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439086+0000", 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014553999999999999 2024-09-15T02:49:38.827 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439109+0000", 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3283000000000001e-05 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448063+0000", 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089537490000000004 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448274+0000", 2024-09-15T02:49:38.828 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000210984 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476192+0000", 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027917830000000001 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:689 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 20.319558783000002, 2024-09-15T02:49:38.829 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037724385999999999, 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 689 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.830 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438963+0000", 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5300000000000002e-07 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438968+0000", 2024-09-15T02:49:38.831 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9180000000000002e-06 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438976+0000", 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6470000000000004e-06 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438981+0000", 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9130000000000002e-06 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.832 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439306+0000", 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032486700000000001 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439319+0000", 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2442999999999999e-05 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448872+0000", 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095532159999999998 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.833 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449323+0000", 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00045103599999999998 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476686+0000", 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027363393 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.834 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:690 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 20.319476633000001, 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038145563, 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 690 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.835 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439045+0000", 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2200000000000003e-07 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.836 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439048+0000", 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.76e-06 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439064+0000", 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5336e-05 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439069+0000", 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8849999999999998e-06 2024-09-15T02:49:38.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439382+0000", 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031364799999999998 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439397+0000", 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4661000000000001e-05 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449673+0000", 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010276298 2024-09-15T02:49:38.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449816+0000", 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014270799999999999 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.477190+0000", 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027373544999999999 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.839 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDScrubReserve(3.19s4 RELEASE e75)", 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 18.476485743000001, 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98297010699999998, 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.840 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282036+0000", 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2270000000000001e-06 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282044+0000", 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6720000000000004e-06 2024-09-15T02:49:38.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264750+0000", 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98270659599999999 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264763+0000", 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2466000000000001e-05 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264912+0000", 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000149524 2024-09-15T02:49:38.842 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264954+0000", 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1461000000000002e-05 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.265005+0000", 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1161e-05 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.843 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s4 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.224208216000001, 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018239887999999999, 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.844 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534314+0000", 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7549999999999999e-06 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534957+0000", 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00064235200000000001 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.845 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535041+0000", 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4050999999999999e-05 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535056+0000", 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5588999999999999e-05 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535177+0000", 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120925 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535207+0000", 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9864000000000001e-05 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552499+0000", 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017291890000000001 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552552+0000", 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3461999999999997e-05 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.bs4 77/76 [PushOp(3:d394ce14:::smithi19125844-2:head, version: 71'77, data_included: [262144~241664], data_size: 241664, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d394ce14:::smithi19125844-2:head@71'77, size: 2015232, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2015232, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 17.122074802, 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.24636945099999999, 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636447+0000", 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.637e-06 2024-09-15T02:49:38.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636697+0000", 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024920600000000003 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867157+0000", 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.23046036 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867176+0000", 2024-09-15T02:49:38.850 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.929e-05 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867332+0000", 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015594099999999999 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.882815+0000", 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015483017 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.851 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.261034240000001, 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025018460999999999, 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.852 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497487+0000", 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8000000000000002e-07 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499654+0000", 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00216738 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.853 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499711+0000", 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7398999999999997e-05 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499723+0000", 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2019e-05 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499830+0000", 2024-09-15T02:49:38.854 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000106718 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499850+0000", 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9732e-05 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522437+0000", 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022587494 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.855 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522505+0000", 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7439000000000003e-05 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 93/76 ECSubWrite(tid=156, reqid=client.4526.0:952, at_version=93'25, trim_to=0'0, roll_forward_to=85'23))", 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 1.472384009, 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038822470999999997, 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:38.856 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:38.857 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4300000000000003e-07 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292226+0000", 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060890609999999998 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292292+0000", 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6272000000000005e-05 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:38.858 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292303+0000", 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0111e-05 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292409+0000", 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010671 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292423+0000", 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3424999999999999e-05 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.859 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.324890+0000", 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.032467754000000001 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.324959+0000", 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8595000000000003e-05 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:38.860 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:38.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:38.863 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:38.858+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:38.963 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:38.956+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:38.970 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:38.970 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:38.970 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:38.970 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:38.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:39.064 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:39.062+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:39.077 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.078 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:39.078 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:39.078 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:39.078 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:39.165 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:39.162+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:39.192 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.619866565000002, 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:49:39.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:39.197 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.609924259000003, 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0099046280000000004, 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579384+0000", 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579386+0000", 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.536e-06 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579616+0000", 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230366 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579660+0000", 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4489e-05 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579668+0000", 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4800000000000004e-06 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579731+0000", 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2733999999999999e-05 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.579742+0000", 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589278+0000", 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095358219999999994 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589289+0000", 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0596e-05 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.201 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 48.538919882999998, 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015141226000000001, 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:39.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650389+0000", 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650600+0000", 2024-09-15T02:49:39.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021155999999999999 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650679+0000", 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8641999999999997e-05 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650693+0000", 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4294000000000001e-05 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650801+0000", 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108132 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650828+0000", 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6565999999999999e-05 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665503+0000", 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014675142 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665530+0000", 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6509999999999999e-05 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.496825649000002, 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011398508, 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.206 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.22e-07 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693204+0000", 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072156 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693258+0000", 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3134999999999999e-05 2024-09-15T02:49:39.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693266+0000", 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5259999999999993e-06 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693333+0000", 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6580000000000003e-05 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693344+0000", 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0971999999999999e-05 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703854+0000", 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010510265 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703881+0000", 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7248000000000001e-05 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.934995020000002, 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016572159999999999, 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254314+0000", 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1100000000000002e-07 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255086+0000", 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077269100000000005 2024-09-15T02:49:39.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255147+0000", 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1005999999999998e-05 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255166+0000", 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8581999999999999e-05 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255267+0000", 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100973 2024-09-15T02:49:39.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255289+0000", 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2565000000000002e-05 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270855+0000", 2024-09-15T02:49:39.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015565733999999999 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270885+0000", 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0298e-05 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.920504065000003, 2024-09-15T02:49:39.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018797873, 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8999999999999998e-07 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269160+0000", 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035577800000000001 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3349000000000001e-05 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269211+0000", 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9439999999999999e-06 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269283+0000", 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2263999999999998e-05 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269296+0000", 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3145e-05 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287519+0000", 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018222955999999998 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287602+0000", 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3146999999999995e-05 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.219 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.912231439000003, 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010563233999999999, 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:39.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277077+0000", 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.277078+0000", 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1500000000000004e-07 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278860+0000", 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017825339999999999 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278958+0000", 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7705000000000001e-05 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.278971+0000", 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3481e-05 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279037+0000", 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5848999999999999e-05 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.279045+0000", 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6359999999999994e-06 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287611+0000", 2024-09-15T02:49:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085663470000000002 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287640+0000", 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8966999999999999e-05 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.as3 55/33 ECSubWrite(tid=36, reqid=client.4526.0:472, at_version=55'21, trim_to=0'0, roll_forward_to=55'19))", 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.261496995000002, 2024-09-15T02:49:39.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010079322999999999, 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927811+0000", 2024-09-15T02:49:39.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.927813+0000", 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9630000000000001e-06 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928385+0000", 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057185600000000004 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928464+0000", 2024-09-15T02:49:39.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8615999999999994e-05 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928480+0000", 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6696000000000001e-05 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928602+0000", 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121209 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.928622+0000", 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.052e-05 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937813+0000", 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091904280000000005 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.937891+0000", 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8034999999999994e-05 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.228 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(3.1s2 RELEASE e76)", 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.519673525999998, 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43008043499999998, 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669637+0000", 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.917e-06 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669648+0000", 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1653e-05 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099516+0000", 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42986811200000002 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099534+0000", 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7997000000000001e-05 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099653+0000", 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011822100000000001 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099681+0000", 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8496999999999998e-05 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099715+0000", 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4038000000000002e-05 2024-09-15T02:49:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.705582541999998, 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012988501, 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483727+0000", 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2500000000000004e-07 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483886+0000", 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000159028 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483939+0000", 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3640999999999997e-05 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483950+0000", 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0462000000000001e-05 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.484771+0000", 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082086099999999999 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.496714+0000", 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011943684 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.236 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.687774218000001, 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010341001000000001, 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501534+0000", 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501535+0000", 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2309999999999999e-06 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501651+0000", 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115538 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501708+0000", 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7479000000000003e-05 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501723+0000", 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4815e-05 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.501805+0000", 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2000000000000001e-05 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511875+0000", 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010069938 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:127fa005:::smithi19125844-18:head, version: 36'10, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:127fa005:::smithi19125844-18:head@36'10, size: 1212416, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.679514253000001, 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012919893999999999, 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.240 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509795+0000", 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8599999999999996e-07 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510077+0000", 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028238699999999999 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510142+0000", 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4518999999999997e-05 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510161+0000", 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8751999999999999e-05 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511930+0000", 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017696719999999999 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.522714+0000", 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010783578 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.243 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.654739757000002, 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0218164, 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6199999999999996e-07 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534737+0000", 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016761400000000001 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534787+0000", 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9836000000000002e-05 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534795+0000", 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1550000000000007e-06 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534859+0000", 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4162e-05 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534875+0000", 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6305000000000001e-05 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556345+0000", 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021469637999999999 2024-09-15T02:49:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556385+0000", 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9827999999999999e-05 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=251, reqid=client.4526.0:761, at_version=77'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.646944991000002, 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014142234, 2024-09-15T02:49:39.248 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542364+0000", 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8700000000000005e-07 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542468+0000", 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010437099999999999 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542522+0000", 2024-09-15T02:49:39.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3295999999999999e-05 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542529+0000", 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1709999999999996e-06 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544564+0000", 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00203476 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544579+0000", 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5605000000000001e-05 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556487+0000", 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011907687 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556506+0000", 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556999999999999e-05 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:18, version: 70'39, data_included: [262144~253952], data_size: 253952, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:18@70'39, size: 2064384, copy_subset: [], clone_subset: {}, snapset: 27=[]:{e=[b],18=[18],27=[27,26,25,24,21,1d]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2064384, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.491086104000001, 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010750387, 2024-09-15T02:49:39.253 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698224+0000", 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2570000000000002e-06 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698584+0000", 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035969000000000002 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698700+0000", 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116018 2024-09-15T02:49:39.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698723+0000", 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3173e-05 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698830+0000", 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010714599999999999 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.708973+0000", 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010142103 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 81/76 ECSubRead(tid=114, to_read={3:517ce0e6:::smithi19125844-47:head=0,475136,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.700198596, 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69833002899999996, 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489111+0000", 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7269999999999999e-06 2024-09-15T02:49:39.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489134+0000", 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3033000000000001e-05 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187052+0000", 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69791777600000005 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187066+0000", 2024-09-15T02:49:39.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3567000000000001e-05 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187169+0000", 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010303599999999999 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187440+0000", 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027088999999999998 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "replica_scrub(pg: 3.0s3,from:0'0,to:85'34,epoch:86/71,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 8.7718581859999993, 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98343883300000001, 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:39.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9999999999999999e-07 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417455+0000", 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-06 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400723+0000", 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98326817399999999 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400733+0000", 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5929999999999995e-06 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400839+0000", 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106413 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:39.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400855+0000", 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5150000000000001e-05 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400889+0000", 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4353000000000001e-05 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=173, reqid=client.4526.0:901, at_version=88'59, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.9084735249999998, 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019105225999999999, 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281129+0000", 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029418700000000001 2024-09-15T02:49:39.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281252+0000", 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123051 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281268+0000", 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5682999999999999e-05 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281408+0000", 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013955900000000001 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281431+0000", 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2952e-05 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299857+0000", 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018425891 2024-09-15T02:49:39.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299940+0000", 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3423000000000004e-05 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=175, reqid=client.4526.0:903, at_version=88'61, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:39.270 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8871821039999999, 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011439048, 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302127+0000", 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.119e-06 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302328+0000", 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000200834 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302386+0000", 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7895e-05 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302394+0000", 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0770000000000008e-06 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302465+0000", 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1017000000000001e-05 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302481+0000", 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5599000000000001e-05 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313526+0000", 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011045743 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313565+0000", 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8763999999999997e-05 2024-09-15T02:49:39.274 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=178, reqid=client.4526.0:905, at_version=0'0, trim_to=0'0, roll_forward_to=88'63))", 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.8695667470000004, 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15873490300000001, 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319766+0000", 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3816e-05 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476980+0000", 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.157214144 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476993+0000", 2024-09-15T02:49:39.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2286000000000001e-05 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477134+0000", 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141006 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477167+0000", 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3847000000000003e-05 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478412+0000", 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012443930000000001 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478476+0000", 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4579999999999995e-05 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.279 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.281 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:39.276+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:39.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:39.283 INFO:tasks.rados.rados.0.smithi191.stdout:380: done (1 left) 2024-09-15T02:49:39.283 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 39 current snap is 38 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:381: seq_num 137 ranges {1933312=753664,2686976=786432,3473408=655360,4128768=114688} 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:381: writing smithi19125844-39 from 1933312 to 2686976 tid 1 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:381: writing smithi19125844-39 from 2686976 to 3473408 tid 2 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:381: writing smithi19125844-39 from 3473408 to 4128768 tid 3 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:381: writing smithi19125844-39 from 4128768 to 4243456 tid 4 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:382: read oid 47 snap -1 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:382: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:383: read oid 30 snap 37 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:383: expect (ObjNum 117 snap 23 seq_num 117) 2024-09-15T02:49:39.284 INFO:tasks.rados.rados.0.smithi191.stdout:384: snap_remove snap 20 2024-09-15T02:49:39.382 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:39.380+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:39.382 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.382 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:39.382 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:39.383 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:39.482 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:39.480+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:39.488 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:39.488 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:39.489 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:39.489 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:39.583 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:39.580+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:39.595 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.023494986999999, 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:49:39.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:49:39.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:39.600 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.010978801999997, 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:49:39.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:49:39.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:49:39.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.009939641999999, 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.606 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:49:39.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:49:39.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:39.611 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 48.942188635999997, 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015031548, 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651048+0000", 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4900000000000001e-07 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651266+0000", 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000218233 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651324+0000", 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8544000000000001e-05 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651335+0000", 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336000000000001e-05 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651408+0000", 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3415000000000001e-05 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651424+0000", 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6001000000000001e-05 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666045+0000", 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014620861000000001 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666079+0000", 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3809000000000001e-05 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.900816251999998, 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015570855999999999, 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692736+0000", 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031561799999999999 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692809+0000", 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3174000000000001e-05 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692821+0000", 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1627000000000001e-05 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692906+0000", 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5172000000000004e-05 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692920+0000", 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4361e-05 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707954+0000", 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015033451999999999 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707991+0000", 2024-09-15T02:49:39.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6714000000000001e-05 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.843754097000001, 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014530774999999999, 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749483+0000", 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4799999999999997e-07 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749717+0000", 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000234408 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749768+0000", 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1155000000000001e-05 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749777+0000", 2024-09-15T02:49:39.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1609999999999992e-06 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749853+0000", 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5803000000000006e-05 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749872+0000", 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556e-05 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.763971+0000", 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014098934 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.764013+0000", 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1810000000000001e-05 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.338873671000002, 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012056918, 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254363+0000", 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7199999999999997e-07 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254896+0000", 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053267899999999999 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255034+0000", 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013821100000000001 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255066+0000", 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255135+0000", 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8885000000000002e-05 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255148+0000", 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2498e-05 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266383+0000", 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011235187000000001 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266419+0000", 2024-09-15T02:49:39.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6358e-05 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.302099065999997, 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011905962000000001, 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291138+0000", 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291342+0000", 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203403 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291572+0000", 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230084 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291579+0000", 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6990000000000003e-06 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291641+0000", 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2093000000000003e-05 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291650+0000", 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303018+0000", 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011368296 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303043+0000", 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4901000000000001e-05 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.301864676000001, 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011692335, 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.634 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291548+0000", 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000177021 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291673+0000", 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012469299999999999 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291678+0000", 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296939+0000", 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052609120000000004 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296948+0000", 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303051+0000", 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006102433 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303064+0000", 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2948999999999999e-05 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/33 ECSubWrite(tid=212, reqid=client.4526.0:450, at_version=52'22, trim_to=0'0, roll_forward_to=49'21))", 2024-09-15T02:49:39.638 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 43.344776492999998, 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015878961, 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248460+0000", 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249338+0000", 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087768999999999998 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249433+0000", 2024-09-15T02:49:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5333000000000004e-05 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249451+0000", 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8406e-05 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249656+0000", 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020463100000000001 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249685+0000", 2024-09-15T02:49:39.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8943000000000001e-05 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264308+0000", 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014623486999999999 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264338+0000", 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0020999999999999e-05 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.306578747, 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020897651, 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286658+0000", 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.58e-07 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287075+0000", 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041690100000000001 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287318+0000", 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024316499999999999 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287324+0000", 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9580000000000004e-06 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287521+0000", 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197746 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287532+0000", 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366e-05 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307520+0000", 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019987953999999999 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307555+0000", 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5102999999999999e-05 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.306141995000001, 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020477789999999999, 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:39.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3200000000000001e-07 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287283+0000", 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018875200000000001 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287367+0000", 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4036000000000007e-05 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287374+0000", 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5340000000000002e-06 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295137+0000", 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077628599999999999 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295149+0000", 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307560+0000", 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012411447000000001 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307572+0000", 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=413, reqid=client.4526.0:673, at_version=72'80, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:39.651 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.300299167999999, 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015088048, 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:39.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8799999999999999e-07 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292992+0000", 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5479000000000002e-05 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293291+0000", 2024-09-15T02:49:39.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029850500000000002 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293295+0000", 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4179999999999999e-06 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302806+0000", 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095105680000000005 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302817+0000", 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1464999999999999e-05 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307993+0000", 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051757540000000003 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308025+0000", 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1671000000000002e-05 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.655 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:c, version: 71'82, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:c@71'82, size: 1081344, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.878550477000001, 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013023088, 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714687+0000", 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.866e-06 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715107+0000", 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419293 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715195+0000", 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8758999999999995e-05 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715215+0000", 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0143e-05 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715370+0000", 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000154611 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.727709+0000", 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012338416 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.659 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:head, version: 71'83, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:head@71'83, size: 2293760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.607713586999999, 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011824942, 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985524+0000", 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.533e-06 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985735+0000", 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021152599999999999 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985816+0000", 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0516e-05 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985833+0000", 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7413000000000001e-05 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985974+0000", 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141161 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.997347+0000", 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011372793000000001 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7f1f21a6:::smithi19125844-10:head, version: 71'85, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:7f1f21a6:::smithi19125844-10:head@71'85, size: 1736704, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.348617039000001, 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014391049, 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.663 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:39.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244620+0000", 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4300000000000001e-06 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245318+0000", 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069816099999999999 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245404+0000", 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5782e-05 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245423+0000", 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8272999999999999e-05 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245544+0000", 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121785 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.259010+0000", 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013465618 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=472, reqid=client.4526.0:793, at_version=79'84, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.109768849999998, 2024-09-15T02:49:39.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013978862999999999, 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483468+0000", 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483595+0000", 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012675800000000001 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483644+0000", 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9200000000000003e-05 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483653+0000", 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4030000000000007e-06 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483747+0000", 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4239000000000004e-05 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483762+0000", 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5698000000000001e-05 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497313+0000", 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013550859 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497446+0000", 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013259400000000001 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.671 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=473, reqid=client.4526.0:794, at_version=79'85, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.103553621, 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014861180999999999, 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:39.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489684+0000", 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.282e-06 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490105+0000", 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042170800000000002 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490163+0000", 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8158000000000001e-05 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490173+0000", 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4350000000000003e-06 2024-09-15T02:49:39.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490301+0000", 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127928 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490314+0000", 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3104e-05 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504495+0000", 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014181545 2024-09-15T02:49:39.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504543+0000", 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8021e-05 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 95/71 ECSubWrite(tid=539, reqid=client.4526.0:958, at_version=95'82, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.32257791400000002, 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025430995000000001, 2024-09-15T02:49:39.676 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270659+0000", 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271073+0000", 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000414342 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271106+0000", 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3107e-05 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271113+0000", 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5910000000000001e-06 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271176+0000", 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3633999999999994e-05 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271187+0000", 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0436e-05 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296049+0000", 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024862042000000001 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296089+0000", 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0102000000000003e-05 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 95/71 ECSubWrite(tid=542, reqid=client.4526.0:959, at_version=95'83, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.31413439999999998, 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017183397, 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:39.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279111+0000", 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.251e-06 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279556+0000", 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044511799999999999 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279593+0000", 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.663e-05 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279600+0000", 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1400000000000002e-06 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279671+0000", 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.161e-05 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279681+0000", 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7559999999999994e-06 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296269+0000", 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016588354 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296285+0000", 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5537999999999999e-05 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:39.685 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:39.686 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.686 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:39.622+0000 7fd7c2dd1540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:49:39.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:39.688 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:39.680+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:39.788 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:39.784+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:39.835 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.836 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:39.836 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:39.836 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:39.889 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:39.887+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:39.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:39.946 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:39.946 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:39.946 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:39.946 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:39.946 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:39.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:39.991 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:39.988+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:40.060 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:40.060 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:40.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 75.692931841999993, 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:40.061 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:40.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:40.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.067 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 75.333360327999998, 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:40.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 74.724497502000006, 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027048119999999998, 2024-09-15T02:49:40.073 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 76 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332180+0000", 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332182+0000", 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5600000000000001e-06 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332190+0000", 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6500000000000002e-06 2024-09-15T02:49:40.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332206+0000", 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5460000000000001e-05 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.332215+0000", 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1800000000000002e-06 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347018+0000", 2024-09-15T02:49:40.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014802801000000001 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347129+0000", 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011114400000000001 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.347300+0000", 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017092400000000001 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:40.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.352562+0000", 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005262035 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.358997+0000", 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064354670000000003 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359137+0000", 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140044 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.359228+0000", 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0854999999999997e-05 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 74.487803884000002, 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.079 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 74.470285308000001, 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.085 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:40.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:40.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.656422880999997, 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:40.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:40.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:40.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.095 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 46.350179242999999, 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:40.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:40.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 46.35012141, 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:40.101 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:40.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:40.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:40.106 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:40.107 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 46.350086224000002, 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:40.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:40.112 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 41.614781665000002, 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:40.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:40.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.116 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.776349451999998, 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:40.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:670 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.773055589999998, 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027236222000000001, 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 670 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8800000000000003e-07 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283625+0000", 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3329999999999999e-06 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283630+0000", 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7450000000000001e-06 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283638+0000", 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7349999999999996e-06 2024-09-15T02:49:40.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283962+0000", 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032339899999999998 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284074+0000", 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011208499999999999 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298432+0000", 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014357939 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298517+0000", 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310858+0000", 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012341388999999999 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.125 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.bs2 72/33 ECSubRead(tid=420, to_read={3:d2661d0e:::smithi19125844-43:head=0,4096,0}, subchunks={3:d2661d0e:::smithi19125844-43:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.745874251, 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085265647999999999, 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310804+0000", 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310814+0000", 2024-09-15T02:49:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0115000000000001e-05 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395542+0000", 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.084728263999999998 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395552+0000", 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5899999999999997e-06 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395608+0000", 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5832999999999999e-05 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.396069+0000", 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046113399999999998 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:753 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.595925778000002, 2024-09-15T02:49:40.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027324345, 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 753 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460754+0000", 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9709999999999998e-06 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460760+0000", 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7510000000000002e-06 2024-09-15T02:49:40.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460773+0000", 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2136999999999999e-05 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460780+0000", 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471867+0000", 2024-09-15T02:49:40.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011087001000000001 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471873+0000", 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000002e-06 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.484980+0000", 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0131073370000001 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485369+0000", 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038873500000000002 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488076+0000", 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027074099999999999 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:754 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:40.134 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.595890189999999, 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0276021209999999, 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 754 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:40.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460788+0000", 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8899999999999999e-07 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460791+0000", 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.411e-06 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460798+0000", 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4679999999999997e-06 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460802+0000", 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2889999999999999e-06 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471896+0000", 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011093591 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471897+0000", 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.331e-06 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485874+0000", 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0139766539999999 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485962+0000", 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8481999999999998e-05 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488389+0000", 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024273060000000002 2024-09-15T02:49:40.139 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:755 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.595871172999999, 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0311918470000001, 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 755 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460807+0000", 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460809+0000", 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460815+0000", 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1249999999999998e-06 2024-09-15T02:49:40.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460819+0000", 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9670000000000003e-06 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471918+0000", 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098856000000001 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471919+0000", 2024-09-15T02:49:40.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488503+0000", 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0165840150000001 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488618+0000", 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011474199999999999 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491998+0000", 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033803650000000002 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:766 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.595854144, 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0356599440000001, 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 766 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:40.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460824+0000", 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.27e-07 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460831+0000", 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460837+0000", 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460841+0000", 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5889999999999999e-06 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471939+0000", 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098591 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471940+0000", 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488932+0000", 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0169917230000001 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.489075+0000", 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014318299999999999 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.496483+0000", 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074080589999999998 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:767 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.595833085999999, 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.037162342, 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 767 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:40.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460845+0000", 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3200000000000001e-07 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460847+0000", 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.554e-06 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460853+0000", 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020000000000002e-06 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460857+0000", 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0679999999999998e-06 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471961+0000", 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011103755999999999 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471962+0000", 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3600000000000003e-07 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.490940+0000", 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.018977662 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491003+0000", 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3081000000000001e-05 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.498007+0000", 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070041510000000001 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:768 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.595816167999999, 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.038324754, 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:40.155 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 768 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:40.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460862+0000", 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.34e-07 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460864+0000", 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6639999999999998e-06 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460870+0000", 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0859999999999998e-06 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460874+0000", 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.563e-06 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471982+0000", 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011108122999999999 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471983+0000", 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6700000000000003e-07 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492584+0000", 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0206010889999999 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492705+0000", 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012115100000000001 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.499186+0000", 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006481077 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 95/77 ECSubWriteReply(tid=293, last_complete=95'119, committed=1, applied=1))", 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.793286033, 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55831691699999997, 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:40.160 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263392+0000", 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-09-15T02:49:40.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263396+0000", 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0960000000000003e-06 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821430+0000", 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55803327199999997 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821437+0000", 2024-09-15T02:49:40.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8320000000000006e-06 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821583+0000", 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014531099999999999 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821708+0000", 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012571500000000001 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:40.163 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:40.164 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:40.165 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:40.165 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:40.165 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:40.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:40.166 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:40.162+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:40.168 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:40.194 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:40.222 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:40.253 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:40.265 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:40.260+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:40.274 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:40.275 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:40.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:49:40.285 INFO:tasks.rados.rados.0.smithi191.stdout:384: done (4 left) 2024-09-15T02:49:40.285 INFO:tasks.rados.rados.0.smithi191.stdout:385: setattr oid 4 current snap is 38 2024-09-15T02:49:40.285 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:40.286 INFO:tasks.rados.rados.0.smithi191.stdout:386: setattr oid 35 current snap is 38 2024-09-15T02:49:40.286 INFO:tasks.rados.rados.0.smithi191.stdout:379: finishing rollback tid 0 to smithi19125844-24 2024-09-15T02:49:40.286 INFO:tasks.rados.rados.0.smithi191.stdout:379: finishing rollback tid 1 to smithi19125844-24 2024-09-15T02:49:40.287 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 24 v 92 (ObjNum 564664608 snap 21890 seq_num 304) dirty dne 2024-09-15T02:49:40.287 INFO:tasks.rados.rados.0.smithi191.stdout:382: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:49:40.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:40.329 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:40.358 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:40.367 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:40.362+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:40.378 INFO:tasks.rados.rados.0.smithi191.stdout:381: finishing write tid 1 to smithi19125844-39 2024-09-15T02:49:40.378 INFO:tasks.rados.rados.0.smithi191.stdout:381: finishing write tid 2 to smithi19125844-39 2024-09-15T02:49:40.378 INFO:tasks.rados.rados.0.smithi191.stdout:381: finishing write tid 3 to smithi19125844-39 2024-09-15T02:49:40.378 INFO:tasks.rados.rados.0.smithi191.stdout:381: finishing write tid 4 to smithi19125844-39 2024-09-15T02:49:40.378 INFO:tasks.rados.rados.0.smithi191.stdout:381: finishing write tid 5 to smithi19125844-39 2024-09-15T02:49:40.379 INFO:tasks.rados.rados.0.smithi191.stdout:381: finishing write tid 6 to smithi19125844-39 2024-09-15T02:49:40.379 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 39 v 86 (ObjNum 137 snap 38 seq_num 137) dirty exists 2024-09-15T02:49:40.379 INFO:tasks.rados.rados.0.smithi191.stdout:381: left oid 39 (ObjNum 137 snap 38 seq_num 137) 2024-09-15T02:49:40.379 INFO:tasks.rados.rados.0.smithi191.stdout:383: expect (ObjNum 117 snap 23 seq_num 117) 2024-09-15T02:49:40.385 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:40.410 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:40.411 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:40.412 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:40.412 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:40.412 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:40.412 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:40.419 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:40.418+0000 7fd7c2dd1540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T02:49:40.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:40.442 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:40.467 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:40.466+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:40.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:40.471+0000 7fd85f7fe700 0 -- 172.21.15.6:0/3168883919 >> [v2:172.21.15.191:6810/22457,v1:172.21.15.191:6811/22457] conn(0x7fd860195a50 msgr2=0x7fd860197e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:40.509 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T02:49:40.512 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T02:49:40.517 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:40.518 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:40.528 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:40.534 INFO:tasks.rados.rados.0.smithi191.stdout:379: done (5 left) 2024-09-15T02:49:40.534 INFO:tasks.rados.rados.0.smithi191.stdout:381: done (4 left) 2024-09-15T02:49:40.534 INFO:tasks.rados.rados.0.smithi191.stdout:382: done (3 left) 2024-09-15T02:49:40.534 INFO:tasks.rados.rados.0.smithi191.stdout:383: done (2 left) 2024-09-15T02:49:40.534 INFO:tasks.rados.rados.0.smithi191.stdout:387: snap_create 2024-09-15T02:49:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 4 v 123 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T02:49:40.535 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 35 v 4 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-15T02:49:40.569 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:40.568+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:40.637 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:40.637 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:40.647 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:40.669 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:40.663+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:40.770 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:40.768+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:40.772 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:40.772 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:40.773 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:40.770+0000 7fd7c2dd1540 -1 osd.6 74 log_to_monitors true 2024-09-15T02:49:40.781 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:40.869 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:40.869 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:40.871 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:40.868+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:40.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:40.972 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:40.968+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:40.986 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:40.986 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:40.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:41.073 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:41.067+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:41.096 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.096 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:41.173 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:41.170+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:41.204 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.204 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:41.274 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:41.268+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:41.296 INFO:tasks.rados.rados.0.smithi191.stdout:385: done (2 left) 2024-09-15T02:49:41.296 INFO:tasks.rados.rados.0.smithi191.stdout:386: done (1 left) 2024-09-15T02:49:41.296 INFO:tasks.rados.rados.0.smithi191.stdout:387: done (0 left) 2024-09-15T02:49:41.296 INFO:tasks.rados.rados.0.smithi191.stdout:388: copy_from oid 11 from oid 18 current snap is 39 2024-09-15T02:49:41.297 INFO:tasks.rados.rados.0.smithi191.stdout:389: read oid 47 snap -1 2024-09-15T02:49:41.297 INFO:tasks.rados.rados.0.smithi191.stdout:389: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:49:41.297 INFO:tasks.rados.rados.0.smithi191.stdout:390: copy_from oid 43 from oid 7 current snap is 39 2024-09-15T02:49:41.297 INFO:tasks.rados.rados.0.smithi191.stdout:391: copy_from oid 1 from oid 31 current snap is 39 2024-09-15T02:49:41.298 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 19 current snap is 39 2024-09-15T02:49:41.298 INFO:tasks.rados.rados.0.smithi191.stdout:392: seq_num 138 ranges {2441216=770048,3211264=540672,3751936=753664,4505600=262144} 2024-09-15T02:49:41.299 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:49:41.315 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:41.310+0000 7fd7b8b17700 -1 osd.6 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T02:49:41.322 INFO:tasks.rados.rados.0.smithi191.stdout:392: writing smithi19125844-19 from 2441216 to 3211264 tid 1 2024-09-15T02:49:41.323 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.323 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.325 INFO:tasks.rados.rados.0.smithi191.stdout:392: writing smithi19125844-19 from 3211264 to 3751936 tid 2 2024-09-15T02:49:41.329 INFO:tasks.rados.rados.0.smithi191.stdout:392: writing smithi19125844-19 from 3751936 to 4505600 tid 3 2024-09-15T02:49:41.332 INFO:tasks.rados.rados.0.smithi191.stdout:392: writing smithi19125844-19 from 4505600 to 4767744 tid 4 2024-09-15T02:49:41.332 INFO:tasks.rados.rados.0.smithi191.stdout:393: delete oid 27 current snap is 39 2024-09-15T02:49:41.332 INFO:tasks.rados.rados.0.smithi191.stdout:389: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-15T02:49:41.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:41.375 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:41.370+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:388: finishing copy_from racing read to smithi19125844-11 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:388: finishing copy_from to smithi19125844-11 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 11 v 61 (ObjNum 132 snap 35 seq_num 132) dirty exists 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:392: finishing write tid 1 to smithi19125844-19 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:392: finishing write tid 2 to smithi19125844-19 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:391: finishing copy_from racing read to smithi19125844-1 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:392: finishing write tid 3 to smithi19125844-19 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:392: finishing write tid 4 to smithi19125844-19 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:391: finishing copy_from to smithi19125844-1 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 1 v 68 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-09-15T02:49:41.413 INFO:tasks.rados.rados.0.smithi191.stdout:390: finishing copy_from racing read to smithi19125844-43 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:388: done (5 left) 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:389: done (4 left) 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:391: done (3 left) 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:393: done (2 left) 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:394: delete oid 1 current snap is 39 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:390: finishing copy_from to smithi19125844-43 2024-09-15T02:49:41.414 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 43 v 27 (ObjNum 92 snap 15 seq_num 92) dirty exists 2024-09-15T02:49:41.415 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:41.416 INFO:tasks.rados.rados.0.smithi191.stdout:390: done (2 left) 2024-09-15T02:49:41.416 INFO:tasks.rados.rados.0.smithi191.stdout:394: done (1 left) 2024-09-15T02:49:41.416 INFO:tasks.rados.rados.0.smithi191.stdout:append oid 28 current snap is 39 2024-09-15T02:49:41.416 INFO:tasks.rados.rados.0.smithi191.stdout:395: seq_num 141 ranges {2867200=425984,3293184=573440,3866624=425984,4292608=524288,4816896=425984} 2024-09-15T02:49:41.428 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.428 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:41.439 INFO:tasks.rados.rados.0.smithi191.stdout:395: writing smithi19125844-28 from 2867200 to 3293184 tid 1 2024-09-15T02:49:41.442 INFO:tasks.rados.rados.0.smithi191.stdout:395: writing smithi19125844-28 from 3293184 to 3866624 tid 2 2024-09-15T02:49:41.445 INFO:tasks.rados.rados.0.smithi191.stdout:395: writing smithi19125844-28 from 3866624 to 4292608 tid 3 2024-09-15T02:49:41.448 INFO:tasks.rados.rados.0.smithi191.stdout:395: writing smithi19125844-28 from 4292608 to 4816896 tid 4 2024-09-15T02:49:41.451 INFO:tasks.rados.rados.0.smithi191.stdout:395: writing smithi19125844-28 from 4816896 to 5242880 tid 5 2024-09-15T02:49:41.451 INFO:tasks.rados.rados.0.smithi191.stdout:396: rmattr oid 7 current snap is 39 2024-09-15T02:49:41.451 INFO:tasks.rados.rados.0.smithi191.stdout:396: done (2 left) 2024-09-15T02:49:41.452 INFO:tasks.rados.rados.0.smithi191.stdout:397: read oid 35 snap -1 2024-09-15T02:49:41.452 INFO:tasks.rados.rados.0.smithi191.stdout:397: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-15T02:49:41.452 INFO:tasks.rados.rados.0.smithi191.stdout:398: delete oid 25 current snap is 39 2024-09-15T02:49:41.460 INFO:tasks.rados.rados.0.smithi191.stdout:398: done (3 left) 2024-09-15T02:49:41.460 INFO:tasks.rados.rados.0.smithi191.stdout:399: copy_from oid 50 from oid 6 current snap is 39 2024-09-15T02:49:41.460 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 4 2024-09-15T02:49:41.460 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 1 to smithi19125844-28 2024-09-15T02:49:41.475 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:41.472+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:41.478 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 2 to smithi19125844-28 2024-09-15T02:49:41.479 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 3 to smithi19125844-28 2024-09-15T02:49:41.481 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 4 to smithi19125844-28 2024-09-15T02:49:41.491 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 5 to smithi19125844-28 2024-09-15T02:49:41.499 INFO:tasks.rados.rados.0.smithi191.stdout:399: finishing copy_from racing read to smithi19125844-50 2024-09-15T02:49:41.499 INFO:tasks.rados.rados.0.smithi191.stdout:399: finishing copy_from to smithi19125844-50 2024-09-15T02:49:41.499 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 50 v 63 (ObjNum 112 snap 22 seq_num 112) dirty exists 2024-09-15T02:49:41.499 INFO:tasks.rados.rados.0.smithi191.stdout:399: done (3 left) 2024-09-15T02:49:41.499 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 3 2024-09-15T02:49:41.522 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:41.522 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:41.522 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:41.522 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:41.531 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-15T02:49:41.547 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.548 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:41.576 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:41.571+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:41.656 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.656 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:41.677 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:41.676+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:41.778 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:41.771+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:41.813 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.813 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:41.869 INFO:tasks.rados.rados.0.smithi191.stdout:397: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-15T02:49:41.880 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:41.876+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:41.896 INFO:tasks.rados.rados.0.smithi191.stdout:397: done (2 left) 2024-09-15T02:49:41.896 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 2 2024-09-15T02:49:41.926 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:41.926 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:41.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:41.961 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 6 to smithi19125844-28 2024-09-15T02:49:41.965 INFO:tasks.rados.rados.0.smithi191.stdout:395: finishing write tid 7 to smithi19125844-28 2024-09-15T02:49:41.965 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 28 v 91 (ObjNum 141 snap 39 seq_num 141) dirty exists 2024-09-15T02:49:41.965 INFO:tasks.rados.rados.0.smithi191.stdout:395: left oid 28 (ObjNum 141 snap 39 seq_num 141) 2024-09-15T02:49:41.965 INFO:tasks.rados.rados.0.smithi191.stdout:395: done (1 left) 2024-09-15T02:49:41.965 INFO:tasks.rados.rados.0.smithi191.stdout: waiting on 1 2024-09-15T02:49:41.981 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:41.978+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:42.038 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.038 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.047 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:42.082 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:42.080+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:42.146 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.147 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:42.183 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:42.180+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:42.257 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.257 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:42.289 INFO:tasks.rados.rados.0.smithi191.stdout:392: finishing write tid 5 to smithi19125844-19 2024-09-15T02:49:42.292 INFO:tasks.rados.rados.0.smithi191.stdout:392: finishing write tid 6 to smithi19125844-19 2024-09-15T02:49:42.292 INFO:tasks.rados.rados.0.smithi191.stdout:update_object_version oid 19 v 73 (ObjNum 138 snap 39 seq_num 138) dirty exists 2024-09-15T02:49:42.292 INFO:tasks.rados.rados.0.smithi191.stdout:392: left oid 19 (ObjNum 138 snap 39 seq_num 138) 2024-09-15T02:49:42.292 INFO:tasks.rados.rados.0.smithi191.stdout:392: done (0 left) 2024-09-15T02:49:42.302 INFO:tasks.rados.rados.0.smithi191.stderr:0 errors. 2024-09-15T02:49:42.302 INFO:tasks.rados.rados.0.smithi191.stderr: 2024-09-15T02:49:42.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:49:42.312 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:42.308+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:42.383 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.383 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.384 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:42.380+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:42.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:42.485 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:42.484+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:42.489 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.489 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:42.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:42.587 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:42.584+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:42.595 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.595 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:42.606 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:49:42.606 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:49:42.289183+0000","last_up_change":"2024-09-15T02:49:42.289183+0000","last_in_change":"2024-09-15T02:49:20.434232+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-15T02:48:36.521842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"71","auid":0,"snap_mode":"selfmanaged","snap_seq":61,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":0,"weight":0,"primary_affinity":0.527313232421875,"last_clean_begin":22,"last_clean_end":74,"up_from":98,"up_thru":47,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":34477},{"type":"v1","addr":"172.21.15.152:6801","nonce":34477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":34477},{"type":"v1","addr":"172.21.15.152:6803","nonce":34477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":34477},{"type":"v1","addr":"172.21.15.152:6807","nonce":34477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":34477},{"type":"v1","addr":"172.21.15.152:6805","nonce":34477}]},"public_addr":"172.21.15.152:6801/34477","cluster_addr":"172.21.15.152:6803/34477","heartbeat_back_addr":"172.21.15.152:6807/34477","heartbeat_front_addr":"172.21.15.152:6805/34477","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":0.2740936279296875,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":29,"up_from":42,"up_thru":76,"down_at":31,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":27185},{"type":"v1","addr":"172.21.15.152:6812","nonce":27185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":27185},{"type":"v1","addr":"172.21.15.152:6819","nonce":27185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":27185},{"type":"v1","addr":"172.21.15.152:6831","nonce":27185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":27185},{"type":"v1","addr":"172.21.15.152:6826","nonce":27185}]},"public_addr":"172.21.15.152:6812/27185","cluster_addr":"172.21.15.152:6819/27185","heartbeat_back_addr":"172.21.15.152:6831/27185","heartbeat_front_addr":"172.21.15.152:6826/27185","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.496434+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:34.269618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:39.185164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.093348+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.890794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.173357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-15T02:48:20.315226+0000","dead_epoch":75},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.289669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:36.555828+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.339470+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:29.360419+0000","dead_epoch":31},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.224457+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.297229+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.117214+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:28.670589+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:48:20.554724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":37,"length":1},{"begin":40,"length":1},{"begin":56,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:49:42.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:49:42.655 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:42.685 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:42.688 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:42.684+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:42.705 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:42.705 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:42.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:42.714 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:42.743 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:42.770 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:42.788 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:42.784+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:42.796 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:42.811 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:42.811 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:42.811 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:42.811 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:42.811 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:42.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:42.889 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:42.883+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:42.924 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:49:42.927 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:42.927 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:42.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:42.927 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 40/33 ECSubWrite(tid=99, reqid=client.4526.0:244, at_version=40'28, trim_to=0'0, roll_forward_to=39'26))", 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 58.341108822999999, 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014940116999999999, 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:42.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577944+0000", 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.577945+0000", 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.037e-06 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578296+0000", 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035186899999999999 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578459+0000", 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016206200000000001 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578482+0000", 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3070000000000001e-05 2024-09-15T02:49:42.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578643+0000", 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016161999999999999 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.578670+0000", 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592839+0000", 2024-09-15T02:49:42.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014168557 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:44.592884+0000", 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4786000000000001e-05 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/33 ECSubWrite(tid=124, reqid=client.4526.0:294, at_version=45'44, trim_to=0'0, roll_forward_to=44'42))", 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:42.932 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.284846688000002, 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016865508000000001, 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634206+0000", 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0999999999999999e-07 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634501+0000", 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029489499999999999 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634605+0000", 2024-09-15T02:49:42.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103488 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634624+0000", 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9530000000000001e-05 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634715+0000", 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1284000000000002e-05 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.634741+0000", 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5350999999999998e-05 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651036+0000", 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016295245999999999 2024-09-15T02:49:42.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:49.651071+0000", 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5203999999999997e-05 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/33 ECSubWrite(tid=180, reqid=client.4526.0:385, at_version=49'53, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.159451355999998, 2024-09-15T02:49:42.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010517344, 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759601+0000", 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759603+0000", 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7179999999999999e-06 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759826+0000", 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0002231 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.759971+0000", 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014553100000000001 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760158+0000", 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018628199999999999 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760305+0000", 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014718500000000001 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.760353+0000", 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8418999999999998e-05 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770084+0000", 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097306449999999996 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770118+0000", 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4464000000000001e-05 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=61, reqid=client.4526.0:417, at_version=50'5, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.634612359000002, 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015418308, 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284440+0000", 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284441+0000", 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.251e-06 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284558+0000", 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011635 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284645+0000", 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6879000000000006e-05 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284653+0000", 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8100000000000004e-06 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284760+0000", 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106585 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284774+0000", 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4260000000000001e-05 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299827+0000", 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015052345 2024-09-15T02:49:42.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299858+0000", 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1828000000000001e-05 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=62, reqid=client.4526.0:418, at_version=50'6, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.634489756999997, 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015317982000000001, 2024-09-15T02:49:42.946 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284563+0000", 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284595+0000", 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2381e-05 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284673+0000", 2024-09-15T02:49:42.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7855000000000006e-05 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284681+0000", 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9910000000000002e-06 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289497+0000", 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048157060000000003 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289509+0000", 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.166e-05 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299868+0000", 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010359454000000001 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299881+0000", 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2622e-05 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 50/33 ECSubWrite(tid=63, reqid=client.4526.0:419, at_version=50'7, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.629540274, 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014955008000000001, 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:42.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289512+0000", 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289513+0000", 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.569999999999999e-07 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289540+0000", 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6984000000000001e-05 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289594+0000", 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4117999999999999e-05 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.289608+0000", 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4219999999999999e-05 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299620+0000", 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010011978 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299633+0000", 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2564e-05 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304436+0000", 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048034269999999999 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.304467+0000", 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0759999999999997e-05 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.628089377999999, 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014581759999999999, 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290963+0000", 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.290964+0000", 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6499999999999998e-07 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291188+0000", 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224513 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291233+0000", 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4665e-05 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291241+0000", 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8019999999999992e-06 2024-09-15T02:49:42.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291323+0000", 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2284e-05 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291336+0000", 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2906e-05 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305516+0000", 2024-09-15T02:49:42.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014179916000000001 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305545+0000", 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8809000000000001e-05 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:42.960 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.627701278000004, 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014221952, 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291351+0000", 2024-09-15T02:49:42.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291352+0000", 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6000000000000004e-07 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291563+0000", 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000211387 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291616+0000", 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3294000000000002e-05 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.291625+0000", 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8559999999999996e-06 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299094+0000", 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007469084 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299105+0000", 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0226e-05 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305552+0000", 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064471270000000004 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.305573+0000", 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1418000000000001e-05 2024-09-15T02:49:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=53, reqid=client.4526.0:536, at_version=60'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.503973793, 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024987760000000001, 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.415079+0000", 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7300000000000001e-07 2024-09-15T02:49:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420805+0000", 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057259099999999999 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420899+0000", 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4377999999999997e-05 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.420915+0000", 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5883e-05 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.421162+0000", 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024698199999999997 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.421192+0000", 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9342000000000001e-05 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440023+0000", 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018831260999999998 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440066+0000", 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3531e-05 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/47 ECSubWrite(tid=288, reqid=client.4526.0:542, at_version=60'32, trim_to=0'0, roll_forward_to=48'29))", 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.494453710000002, 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011886391, 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424599+0000", 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424601+0000", 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1330000000000002e-06 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424662+0000", 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1493000000000002e-05 2024-09-15T02:49:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424768+0000", 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010543 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424786+0000", 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.876e-05 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424886+0000", 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9922999999999999e-05 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.424906+0000", 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9110000000000002e-05 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436423+0000", 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011517135 2024-09-15T02:49:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.436485+0000", 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2407000000000001e-05 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/33 ECSubWrite(tid=54, reqid=client.4526.0:537, at_version=60'7, trim_to=0'0, roll_forward_to=36'4))", 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.493551263999997, 2024-09-15T02:49:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014882912, 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425501+0000", 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425503+0000", 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425746+0000", 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024368799999999999 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425834+0000", 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7188999999999995e-05 2024-09-15T02:49:42.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.425848+0000", 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4107000000000001e-05 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429379+0000", 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035311700000000001 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429394+0000", 2024-09-15T02:49:42.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5582000000000001e-05 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440351+0000", 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010956702 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440384+0000", 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2873999999999997e-05 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/47 ECSubWrite(tid=289, reqid=client.4526.0:543, at_version=60'33, trim_to=0'0, roll_forward_to=48'29))", 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.490527497999999, 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011573594, 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428525+0000", 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428526+0000", 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.482e-06 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428559+0000", 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2526e-05 2024-09-15T02:49:42.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428636+0000", 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.674e-05 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.428655+0000", 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9664999999999999e-05 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429356+0000", 2024-09-15T02:49:42.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070097400000000004 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.429370+0000", 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3755e-05 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440078+0000", 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010708255999999999 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:04.440098+0000", 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0196000000000001e-05 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.15s1 77/76 [PushOp(3:a906e5df:::smithi19125844-36:c, version: 72'52, data_included: [0~147456], data_size: 147456, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:a906e5df:::smithi19125844-36:c@72'52, size: 589824, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:589824, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.394602+0000", 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.524450676000001, 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010446046, 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394602+0000", 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394602+0000", 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394603+0000", 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6700000000000001e-06 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394838+0000", 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023451799999999999 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394925+0000", 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.394948+0000", 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3206000000000001e-05 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.395097+0000", 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014951999999999999 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.405048+0000", 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099504439999999993 2024-09-15T02:49:42.986 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.15s1 77/76 [PushOp(3:a906e5df:::smithi19125844-36:head, version: 72'53, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a906e5df:::smithi19125844-36:head@72'53, size: 1835008, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.411603+0000", 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.507449415, 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013109686000000001, 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411603+0000", 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411603+0000", 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411604+0000", 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-15T02:49:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411872+0000", 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000268299 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411956+0000", 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4110999999999995e-05 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.411979+0000", 2024-09-15T02:49:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2889999999999999e-05 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.412117+0000", 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013782399999999999 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.424713+0000", 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012595653 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s2 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 23, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.568702+0000", 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.350350249000002, 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013310231, 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.568702+0000", 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.568702+0000", 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.568704+0000", 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569645+0000", 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094167199999999995 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569735+0000", 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9248999999999999e-05 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569758+0000", 2024-09-15T02:49:42.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569898+0000", 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001393 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.582012+0000", 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012114860999999999 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s2 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.773456+0000", 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.145596150999999, 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013906271, 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773456+0000", 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773456+0000", 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773458+0000", 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8500000000000001e-06 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774079+0000", 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062117800000000001 2024-09-15T02:49:42.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774123+0000", 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3902000000000001e-05 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774136+0000", 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3055e-05 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774351+0000", 2024-09-15T02:49:42.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021454300000000001 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.787363+0000", 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013011742999999999 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 83/79 ECSubWrite(tid=486, reqid=client.4526.0:823, at_version=83'105, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.548804+0000", 2024-09-15T02:49:42.998 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.370247944000001, 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095958910000000005, 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548804+0000", 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548804+0000", 2024-09-15T02:49:42.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.548806+0000", 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.646e-06 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549034+0000", 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022757300000000001 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549110+0000", 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6501999999999994e-05 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549123+0000", 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2765999999999999e-05 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549221+0000", 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7750000000000004e-05 2024-09-15T02:49:43.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.549244+0000", 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2900000000000001e-05 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.558366+0000", 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091223180000000008 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.558400+0000", 2024-09-15T02:49:43.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4436e-05 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 83/79 ECSubWrite(tid=487, reqid=client.4526.0:824, at_version=83'106, trim_to=0'0, roll_forward_to=83'104))", 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.560882+0000", 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.358170094, 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099151139999999992, 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.560882+0000", 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.560882+0000", 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.560884+0000", 2024-09-15T02:49:43.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0090000000000002e-06 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561033+0000", 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014894800000000001 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561189+0000", 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015607699999999999 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561232+0000", 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2800999999999999e-05 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561348+0000", 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115623 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.561356+0000", 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2679999999999994e-06 2024-09-15T02:49:43.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570768+0000", 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094117799999999998 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.570797+0000", 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9607999999999998e-05 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es6 83/76 ECSubWrite(tid=130, reqid=client.4526.0:837, at_version=83'97, trim_to=0'0, roll_forward_to=83'95))", 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:27.623549+0000", 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.295503061, 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012980457000000001, 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.623549+0000", 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.623549+0000", 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.623550+0000", 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1400000000000001e-06 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624037+0000", 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048631199999999997 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624120+0000", 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3255000000000002e-05 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624154+0000", 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3921999999999998e-05 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624234+0000", 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0131999999999998e-05 2024-09-15T02:49:43.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.624247+0000", 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2692e-05 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.636483+0000", 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012236629000000001 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:27.636530+0000", 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6375000000000002e-05 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.15s1 88/76 ECSubWrite(tid=175, reqid=client.4526.0:903, at_version=88'61, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.302753+0000", 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.616299179, 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095794560000000001, 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.302753+0000", 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.302753+0000", 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.302754+0000", 2024-09-15T02:49:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.28e-06 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303118+0000", 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036316400000000002 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303181+0000", 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3756000000000004e-05 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303192+0000", 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0783e-05 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303258+0000", 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5732e-05 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.303266+0000", 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6369999999999999e-06 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.312286+0000", 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090208379999999998 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.312333+0000", 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6266000000000001e-05 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.017 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.019 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:49:43.019 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:43.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:43.021 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-15T02:49:42.293610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125248051,"num_objects":119,"num_object_clones":67,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":8,"num_read":375,"num_read_kb":146452,"num_write":680,"num_write_kb":95802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":45481984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1709,"ondisk_log_size":1709,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":4711536,"kb_used_data":221480,"kb_used_omap":25,"kb_used_meta":4489766,"kb_avail":1410866064,"statfs":{"total":1449551462400,"available":1444726849536,"internally_reserved":0,"allocated":226795520,"data_stored":303300682,"data_compressed":2097709,"data_compressed_allocated":4407296,"data_compressed_original":9707520,"omap_allocated":25640,"internal_metadata":4597521368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043380"},"pg_stats":[{"pgid":"3.19","version":"92'61","reported_seq":126,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T02:49:37.291323+0000","last_change":"2024-09-15T02:49:37.291382+0000","last_active":"2024-09-15T02:49:37.291323+0000","last_peered":"2024-09-15T02:49:37.291323+0000","last_clean":"2024-09-15T02:49:37.291323+0000","last_became_active":"2024-09-15T02:49:18.470335+0000","last_became_peered":"2024-09-15T02:49:18.470335+0000","last_unstale":"2024-09-15T02:49:37.291323+0000","last_undegraded":"2024-09-15T02:49:37.291323+0000","last_fullsized":"2024-09-15T02:49:37.291323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'51","last_scrub_stamp":"2024-09-15T02:49:21.298540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:21.298540+0000","objects_scrubbed":5,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:25.010610+0000","scrub_duration":0.0083571319999999998,"objects_trimmed":0,"snaptrim_duration":0.015845198000000001,"stat_sum":{"num_bytes":5652480,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,0,3,14,8,11],"acting":[9,1,0,3,14,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"b","length":"1"}]},{"pgid":"3.18","version":"87'62","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.245222+0000","last_change":"2024-09-15T02:49:37.246494+0000","last_active":"2024-09-15T02:49:37.245222+0000","last_peered":"2024-09-15T02:49:37.245222+0000","last_clean":"2024-09-15T02:49:37.245222+0000","last_became_active":"2024-09-15T02:49:21.472717+0000","last_became_peered":"2024-09-15T02:49:21.472717+0000","last_unstale":"2024-09-15T02:49:37.245222+0000","last_undegraded":"2024-09-15T02:49:37.245222+0000","last_fullsized":"2024-09-15T02:49:37.245222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'56","last_scrub_stamp":"2024-09-15T02:49:29.290881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.290881+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:47.579564+0000","scrub_duration":0.0059159260000000002,"objects_trimmed":0,"snaptrim_duration":0.0016002189999999999,"stat_sum":{"num_bytes":4227072,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1,13,11,12,5],"acting":[3,9,1,13,11,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.17","version":"90'117","reported_seq":200,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.256934+0000","last_change":"2024-09-15T02:49:37.247935+0000","last_active":"2024-09-15T02:49:37.256934+0000","last_peered":"2024-09-15T02:49:37.256934+0000","last_clean":"2024-09-15T02:49:37.256934+0000","last_became_active":"2024-09-15T02:49:22.482618+0000","last_became_peered":"2024-09-15T02:49:22.482618+0000","last_unstale":"2024-09-15T02:49:37.256934+0000","last_undegraded":"2024-09-15T02:49:37.256934+0000","last_fullsized":"2024-09-15T02:49:37.256934+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028460730000000002,"stat_sum":{"num_bytes":3563520,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":25538,"num_write":46,"num_write_kb":5232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6193152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12,10,7,4,15,3],"acting":[11,12,10,7,4,15,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.16","version":"85'34","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.289066+0000","last_change":"2024-09-15T02:49:37.289810+0000","last_active":"2024-09-15T02:49:37.289066+0000","last_peered":"2024-09-15T02:49:37.289066+0000","last_clean":"2024-09-15T02:49:37.289066+0000","last_became_active":"2024-09-15T02:49:18.462615+0000","last_became_peered":"2024-09-15T02:49:18.462615+0000","last_unstale":"2024-09-15T02:49:37.289066+0000","last_undegraded":"2024-09-15T02:49:37.289066+0000","last_fullsized":"2024-09-15T02:49:37.289066+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'34","last_scrub_stamp":"2024-09-15T02:49:30.409672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:30.409672+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:48.545363+0000","scrub_duration":0.0056793069999999998,"objects_trimmed":0,"snaptrim_duration":0.001774858,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,11,9,5,7,15,10],"acting":[12,11,9,5,7,15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.15","version":"88'63","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260730+0000","last_change":"2024-09-15T02:49:37.260730+0000","last_active":"2024-09-15T02:49:37.260730+0000","last_peered":"2024-09-15T02:49:37.260730+0000","last_clean":"2024-09-15T02:49:37.260730+0000","last_became_active":"2024-09-15T02:49:21.482421+0000","last_became_peered":"2024-09-15T02:49:21.482421+0000","last_unstale":"2024-09-15T02:49:37.260730+0000","last_undegraded":"2024-09-15T02:49:37.260730+0000","last_fullsized":"2024-09-15T02:49:37.260730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'57","last_scrub_stamp":"2024-09-15T02:49:30.515515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:30.515515+0000","objects_scrubbed":5,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:32.152375+0000","scrub_duration":0.58297202400000003,"objects_trimmed":0,"snaptrim_duration":0.0016319049999999999,"stat_sum":{"num_bytes":6864896,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5,8,7,4,2,3],"acting":[15,5,8,7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.14","version":"36'2","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260543+0000","last_change":"2024-09-15T02:49:37.261987+0000","last_active":"2024-09-15T02:49:37.260543+0000","last_peered":"2024-09-15T02:49:37.260543+0000","last_clean":"2024-09-15T02:49:37.260543+0000","last_became_active":"2024-09-15T02:49:18.463893+0000","last_became_peered":"2024-09-15T02:49:18.463893+0000","last_unstale":"2024-09-15T02:49:37.260543+0000","last_undegraded":"2024-09-15T02:49:37.260543+0000","last_fullsized":"2024-09-15T02:49:37.260543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'2","last_scrub_stamp":"2024-09-15T02:49:32.931133+0000","last_deep_scrub":"36'2","last_deep_scrub_stamp":"2024-09-15T02:49:32.931133+0000","last_clean_scrub_stamp":"2024-09-15T02:49:32.931133+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:59.917241+0000","scrub_duration":0.222807329,"objects_trimmed":0,"snaptrim_duration":0.0016873490000000001,"stat_sum":{"num_bytes":573440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,12,15,11,9,13],"acting":[4,7,12,15,11,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260677+0000","last_change":"2024-09-15T02:48:52.652454+0000","last_active":"2024-09-15T02:49:37.260677+0000","last_peered":"2024-09-15T02:49:37.260677+0000","last_clean":"2024-09-15T02:49:37.260677+0000","last_became_active":"2024-09-15T02:48:52.652176+0000","last_became_peered":"2024-09-15T02:48:52.652176+0000","last_unstale":"2024-09-15T02:49:37.260677+0000","last_undegraded":"2024-09-15T02:49:37.260677+0000","last_fullsized":"2024-09-15T02:49:37.260677+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:56.216530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"85'34","reported_seq":239,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.257245+0000","last_change":"2024-09-15T02:49:37.257470+0000","last_active":"2024-09-15T02:49:37.257245+0000","last_peered":"2024-09-15T02:49:37.257245+0000","last_clean":"2024-09-15T02:49:37.257245+0000","last_became_active":"2024-09-15T02:49:16.262927+0000","last_became_peered":"2024-09-15T02:49:16.262927+0000","last_unstale":"2024-09-15T02:49:37.257245+0000","last_undegraded":"2024-09-15T02:49:37.257245+0000","last_fullsized":"2024-09-15T02:49:37.257245+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'32","last_scrub_stamp":"2024-09-15T02:49:17.616353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:17.616353+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:27.488386+0000","scrub_duration":0.0062733370000000004,"objects_trimmed":0,"snaptrim_duration":0.0029848209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.261663+0000","last_change":"2024-09-15T02:49:21.467125+0000","last_active":"2024-09-15T02:49:36.261663+0000","last_peered":"2024-09-15T02:49:36.261663+0000","last_clean":"2024-09-15T02:49:36.261663+0000","last_became_active":"2024-09-15T02:49:21.466842+0000","last_became_peered":"2024-09-15T02:49:21.466842+0000","last_unstale":"2024-09-15T02:49:36.261663+0000","last_undegraded":"2024-09-15T02:49:36.261663+0000","last_fullsized":"2024-09-15T02:49:36.261663+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:39.448131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'117","reported_seq":353,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.257603+0000","last_change":"2024-09-15T02:49:37.257849+0000","last_active":"2024-09-15T02:49:37.257603+0000","last_peered":"2024-09-15T02:49:37.257603+0000","last_clean":"2024-09-15T02:49:37.257603+0000","last_became_active":"2024-09-15T02:49:24.496338+0000","last_became_peered":"2024-09-15T02:49:24.496338+0000","last_unstale":"2024-09-15T02:49:37.257603+0000","last_undegraded":"2024-09-15T02:49:37.257603+0000","last_fullsized":"2024-09-15T02:49:37.257603+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032092259999999999,"stat_sum":{"num_bytes":8314880,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":12876,"num_write":36,"num_write_kb":11184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5259264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,7,1,11,5,0],"acting":[13,4,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.364282+0000","last_change":"2024-09-15T02:49:21.465919+0000","last_active":"2024-09-15T02:49:36.364282+0000","last_peered":"2024-09-15T02:49:36.364282+0000","last_clean":"2024-09-15T02:49:36.364282+0000","last_became_active":"2024-09-15T02:49:21.465616+0000","last_became_peered":"2024-09-15T02:49:21.465616+0000","last_unstale":"2024-09-15T02:49:36.364282+0000","last_undegraded":"2024-09-15T02:49:36.364282+0000","last_fullsized":"2024-09-15T02:49:36.364282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.583189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,13],"acting":[5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"36'2","reported_seq":182,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.245191+0000","last_change":"2024-09-15T02:49:37.245946+0000","last_active":"2024-09-15T02:49:37.245191+0000","last_peered":"2024-09-15T02:49:37.245191+0000","last_clean":"2024-09-15T02:49:37.245191+0000","last_became_active":"2024-09-15T02:49:16.256552+0000","last_became_peered":"2024-09-15T02:49:16.256552+0000","last_unstale":"2024-09-15T02:49:37.245191+0000","last_undegraded":"2024-09-15T02:49:37.245191+0000","last_fullsized":"2024-09-15T02:49:37.245191+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'2","last_scrub_stamp":"2024-09-15T02:49:17.582201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:17.582201+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:31.619423+0000","scrub_duration":0.0051105170000000002,"objects_trimmed":0,"snaptrim_duration":0.001000711,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":87,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.243021+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:49:36.243021+0000","last_peered":"2024-09-15T02:49:36.243021+0000","last_clean":"2024-09-15T02:49:36.243021+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:49:36.243021+0000","last_undegraded":"2024-09-15T02:49:36.243021+0000","last_fullsized":"2024-09-15T02:49:36.243021+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"88'57","reported_seq":271,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.256526+0000","last_change":"2024-09-15T02:49:37.256872+0000","last_active":"2024-09-15T02:49:37.256526+0000","last_peered":"2024-09-15T02:49:37.256526+0000","last_clean":"2024-09-15T02:49:37.256526+0000","last_became_active":"2024-09-15T02:49:21.482725+0000","last_became_peered":"2024-09-15T02:49:21.482725+0000","last_unstale":"2024-09-15T02:49:37.256526+0000","last_undegraded":"2024-09-15T02:49:37.256526+0000","last_fullsized":"2024-09-15T02:49:37.256526+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'57","last_scrub_stamp":"2024-09-15T02:49:34.649254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:34.649254+0000","objects_scrubbed":2,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:42.297214+0000","scrub_duration":0.0069899530000000001,"objects_trimmed":0,"snaptrim_duration":0.0027545500000000001,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,15],"acting":[8,13,5,11,9,1,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.3","version":"85'98","reported_seq":284,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260450+0000","last_change":"2024-09-15T02:49:37.260812+0000","last_active":"2024-09-15T02:49:37.260450+0000","last_peered":"2024-09-15T02:49:37.260450+0000","last_clean":"2024-09-15T02:49:37.260450+0000","last_became_active":"2024-09-15T02:49:16.251895+0000","last_became_peered":"2024-09-15T02:49:16.251895+0000","last_unstale":"2024-09-15T02:49:37.260450+0000","last_undegraded":"2024-09-15T02:49:37.260450+0000","last_fullsized":"2024-09-15T02:49:37.260450+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'90","last_scrub_stamp":"2024-09-15T02:49:21.457038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:21.457038+0000","objects_scrubbed":7,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:25.998414+0000","scrub_duration":0.30840548299999998,"objects_trimmed":0,"snaptrim_duration":0.0014053620000000001,"stat_sum":{"num_bytes":6324224,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":6450,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.261353+0000","last_change":"2024-09-15T02:48:52.654051+0000","last_active":"2024-09-15T02:49:36.261353+0000","last_peered":"2024-09-15T02:49:36.261353+0000","last_clean":"2024-09-15T02:49:36.261353+0000","last_became_active":"2024-09-15T02:48:52.653285+0000","last_became_peered":"2024-09-15T02:48:52.653285+0000","last_unstale":"2024-09-15T02:49:36.261353+0000","last_undegraded":"2024-09-15T02:49:36.261353+0000","last_fullsized":"2024-09-15T02:49:36.261353+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:44.752359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"85'34","reported_seq":226,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260937+0000","last_change":"2024-09-15T02:49:37.261720+0000","last_active":"2024-09-15T02:49:37.260937+0000","last_peered":"2024-09-15T02:49:37.260937+0000","last_clean":"2024-09-15T02:49:37.260937+0000","last_became_active":"2024-09-15T02:49:16.260290+0000","last_became_peered":"2024-09-15T02:49:16.260290+0000","last_unstale":"2024-09-15T02:49:37.260937+0000","last_undegraded":"2024-09-15T02:49:37.260937+0000","last_fullsized":"2024-09-15T02:49:37.260937+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'34","last_scrub_stamp":"2024-09-15T02:49:31.403074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.403074+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:48.299772+0000","scrub_duration":0.99022798000000001,"objects_trimmed":0,"snaptrim_duration":0.0010219719999999999,"stat_sum":{"num_bytes":3276800,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.248604+0000","last_change":"2024-09-15T02:48:29.420976+0000","last_active":"2024-09-15T02:49:36.248604+0000","last_peered":"2024-09-15T02:49:36.248604+0000","last_clean":"2024-09-15T02:49:36.248604+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:49:36.248604+0000","last_undegraded":"2024-09-15T02:49:36.248604+0000","last_fullsized":"2024-09-15T02:49:36.248604+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"85'80","reported_seq":314,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.254287+0000","last_change":"2024-09-15T02:49:37.254783+0000","last_active":"2024-09-15T02:49:37.254287+0000","last_peered":"2024-09-15T02:49:37.254287+0000","last_clean":"2024-09-15T02:49:37.254287+0000","last_became_active":"2024-09-15T02:49:16.255255+0000","last_became_peered":"2024-09-15T02:49:16.255255+0000","last_unstale":"2024-09-15T02:49:37.254287+0000","last_undegraded":"2024-09-15T02:49:37.254287+0000","last_fullsized":"2024-09-15T02:49:37.254287+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'80","last_scrub_stamp":"2024-09-15T02:49:33.693549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:33.693549+0000","objects_scrubbed":5,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:56.782722+0000","scrub_duration":0.011892081000000001,"objects_trimmed":0,"snaptrim_duration":0.000736709,"stat_sum":{"num_bytes":7929856,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-15T02:49:40.288243+0000","last_change":"2024-09-15T02:49:21.466149+0000","last_active":"2024-09-15T02:49:40.288243+0000","last_peered":"2024-09-15T02:49:40.288243+0000","last_clean":"2024-09-15T02:49:40.288243+0000","last_became_active":"2024-09-15T02:49:21.465721+0000","last_became_peered":"2024-09-15T02:49:21.465721+0000","last_unstale":"2024-09-15T02:49:40.288243+0000","last_undegraded":"2024-09-15T02:49:40.288243+0000","last_fullsized":"2024-09-15T02:49:40.288243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"70'37","reported_seq":203,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.248389+0000","last_change":"2024-09-15T02:49:37.249354+0000","last_active":"2024-09-15T02:49:37.248389+0000","last_peered":"2024-09-15T02:49:37.248389+0000","last_clean":"2024-09-15T02:49:37.248389+0000","last_became_active":"2024-09-15T02:49:16.263776+0000","last_became_peered":"2024-09-15T02:49:16.263776+0000","last_unstale":"2024-09-15T02:49:37.248389+0000","last_undegraded":"2024-09-15T02:49:37.248389+0000","last_fullsized":"2024-09-15T02:49:37.248389+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042541480000000001,"stat_sum":{"num_bytes":1982464,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":28,"num_write_kb":2064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.241459+0000","last_change":"2024-09-15T02:48:29.636494+0000","last_active":"2024-09-15T02:49:36.241459+0000","last_peered":"2024-09-15T02:49:36.241459+0000","last_clean":"2024-09-15T02:49:36.241459+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:49:36.241459+0000","last_undegraded":"2024-09-15T02:49:36.241459+0000","last_fullsized":"2024-09-15T02:49:36.241459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":213,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T02:49:36.241618+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:49:36.241618+0000","last_peered":"2024-09-15T02:49:36.241618+0000","last_clean":"2024-09-15T02:49:36.241618+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:49:36.241618+0000","last_undegraded":"2024-09-15T02:49:36.241618+0000","last_fullsized":"2024-09-15T02:49:36.241618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:50.005063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"90'61","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.245006+0000","last_change":"2024-09-15T02:49:37.248362+0000","last_active":"2024-09-15T02:49:37.245006+0000","last_peered":"2024-09-15T02:49:37.245006+0000","last_clean":"2024-09-15T02:49:37.245006+0000","last_became_active":"2024-09-15T02:49:21.477251+0000","last_became_peered":"2024-09-15T02:49:21.477251+0000","last_unstale":"2024-09-15T02:49:37.245006+0000","last_undegraded":"2024-09-15T02:49:37.245006+0000","last_fullsized":"2024-09-15T02:49:37.245006+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033521200000000001,"stat_sum":{"num_bytes":7692288,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":18128,"num_write":43,"num_write_kb":7960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,3],"acting":[11,1,2,15,14,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.9","version":"85'55","reported_seq":290,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260834+0000","last_change":"2024-09-15T02:49:37.261797+0000","last_active":"2024-09-15T02:49:37.260834+0000","last_peered":"2024-09-15T02:49:37.260834+0000","last_clean":"2024-09-15T02:49:37.260834+0000","last_became_active":"2024-09-15T02:49:16.253291+0000","last_became_peered":"2024-09-15T02:49:16.253291+0000","last_unstale":"2024-09-15T02:49:37.260834+0000","last_undegraded":"2024-09-15T02:49:37.260834+0000","last_fullsized":"2024-09-15T02:49:37.260834+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'47","last_scrub_stamp":"2024-09-15T02:49:17.495488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:17.495488+0000","objects_scrubbed":1,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:43.440276+0000","scrub_duration":0.005330581,"objects_trimmed":0,"snaptrim_duration":0.0011959869999999999,"stat_sum":{"num_bytes":442368,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":2594,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.a","version":"93'76","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.295624+0000","last_change":"2024-09-15T02:49:37.296654+0000","last_active":"2024-09-15T02:49:37.295624+0000","last_peered":"2024-09-15T02:49:37.295624+0000","last_clean":"2024-09-15T02:49:37.295624+0000","last_became_active":"2024-09-15T02:49:21.475789+0000","last_became_peered":"2024-09-15T02:49:21.475789+0000","last_unstale":"2024-09-15T02:49:37.295624+0000","last_undegraded":"2024-09-15T02:49:37.295624+0000","last_fullsized":"2024-09-15T02:49:37.295624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:43.485705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081466439999999998,"stat_sum":{"num_bytes":7553024,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":23,"num_read_kb":13641,"num_write":28,"num_write_kb":7824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5849088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,0,8],"acting":[12,4,7,3,14,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.b","version":"93'114","reported_seq":321,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.257312+0000","last_change":"2024-09-15T02:49:37.257312+0000","last_active":"2024-09-15T02:49:37.257312+0000","last_peered":"2024-09-15T02:49:37.257312+0000","last_clean":"2024-09-15T02:49:37.257312+0000","last_became_active":"2024-09-15T02:49:21.476878+0000","last_became_peered":"2024-09-15T02:49:21.476878+0000","last_unstale":"2024-09-15T02:49:37.257312+0000","last_undegraded":"2024-09-15T02:49:37.257312+0000","last_fullsized":"2024-09-15T02:49:37.257312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:04.605427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032064630000000001,"stat_sum":{"num_bytes":15400960,"num_objects":14,"num_object_clones":9,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":42,"num_read_kb":20416,"num_write":91,"num_write_kb":17568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":10584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,14,4,7],"acting":[13,1,11,15,14,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.c","version":"93'25","reported_seq":227,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.336095+0000","last_change":"2024-09-15T02:49:37.258268+0000","last_active":"2024-09-15T02:49:37.336095+0000","last_peered":"2024-09-15T02:49:37.336095+0000","last_clean":"2024-09-15T02:49:37.336095+0000","last_became_active":"2024-09-15T02:49:21.464619+0000","last_became_peered":"2024-09-15T02:49:21.464619+0000","last_unstale":"2024-09-15T02:49:37.336095+0000","last_undegraded":"2024-09-15T02:49:37.336095+0000","last_fullsized":"2024-09-15T02:49:37.336095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:03.879821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022581020000000001,"stat_sum":{"num_bytes":7651328,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":9,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3899392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,14,1,9],"acting":[0,13,12,10,14,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.d","version":"85'25","reported_seq":188,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.289923+0000","last_change":"2024-09-15T02:49:37.290406+0000","last_active":"2024-09-15T02:49:37.289923+0000","last_peered":"2024-09-15T02:49:37.289923+0000","last_clean":"2024-09-15T02:49:37.289923+0000","last_became_active":"2024-09-15T02:48:38.545950+0000","last_became_peered":"2024-09-15T02:48:38.545950+0000","last_unstale":"2024-09-15T02:49:37.289923+0000","last_undegraded":"2024-09-15T02:49:37.289923+0000","last_fullsized":"2024-09-15T02:49:37.289923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:58.729969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001554988,"stat_sum":{"num_bytes":2883584,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":10518,"num_write":10,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.e","version":"91'114","reported_seq":330,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.311392+0000","last_change":"2024-09-15T02:49:37.311779+0000","last_active":"2024-09-15T02:49:37.311392+0000","last_peered":"2024-09-15T02:49:37.311392+0000","last_clean":"2024-09-15T02:49:37.311392+0000","last_became_active":"2024-09-15T02:49:21.473900+0000","last_became_peered":"2024-09-15T02:49:21.473900+0000","last_unstale":"2024-09-15T02:49:37.311392+0000","last_undegraded":"2024-09-15T02:49:37.311392+0000","last_fullsized":"2024-09-15T02:49:37.311392+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:42.136893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057176099999999995,"stat_sum":{"num_bytes":12419072,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":38,"num_read_kb":20233,"num_write":127,"num_write_kb":14800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,7,4,13,0,5],"acting":[2,11,7,4,13,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.f","version":"90'82","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.256286+0000","last_change":"2024-09-15T02:49:37.257108+0000","last_active":"2024-09-15T02:49:37.256286+0000","last_peered":"2024-09-15T02:49:37.256286+0000","last_clean":"2024-09-15T02:49:37.256286+0000","last_became_active":"2024-09-15T02:48:58.322126+0000","last_became_peered":"2024-09-15T02:48:58.322126+0000","last_unstale":"2024-09-15T02:49:37.256286+0000","last_undegraded":"2024-09-15T02:49:37.256286+0000","last_fullsized":"2024-09-15T02:49:37.256286+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:03.904159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010884550000000001,"stat_sum":{"num_bytes":8355840,"num_objects":10,"num_object_clones":6,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":8317,"num_write":52,"num_write_kb":8160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.10","version":"70'30","reported_seq":68,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.260505+0000","last_change":"2024-09-15T02:49:37.260505+0000","last_active":"2024-09-15T02:49:37.260505+0000","last_peered":"2024-09-15T02:49:37.260505+0000","last_clean":"2024-09-15T02:49:37.260505+0000","last_became_active":"2024-09-15T02:49:21.475674+0000","last_became_peered":"2024-09-15T02:49:21.475674+0000","last_unstale":"2024-09-15T02:49:37.260505+0000","last_undegraded":"2024-09-15T02:49:37.260505+0000","last_fullsized":"2024-09-15T02:49:37.260505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010066389999999999,"stat_sum":{"num_bytes":1638400,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":7660,"num_write":6,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8,15,5,9,0],"acting":[7,11,8,15,5,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.11","version":"96'71","reported_seq":98,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-15T02:49:40.308678+0000","last_change":"2024-09-15T02:49:40.308678+0000","last_active":"2024-09-15T02:49:40.308678+0000","last_peered":"2024-09-15T02:49:40.308678+0000","last_clean":"2024-09-15T02:49:40.308678+0000","last_became_active":"2024-09-15T02:49:18.464014+0000","last_became_peered":"2024-09-15T02:49:18.464014+0000","last_unstale":"2024-09-15T02:49:40.308678+0000","last_undegraded":"2024-09-15T02:49:40.308678+0000","last_fullsized":"2024-09-15T02:49:40.308678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'65","last_scrub_stamp":"2024-09-15T02:49:31.195982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.195982+0000","objects_scrubbed":2,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:54.263248+0000","scrub_duration":0.0066166080000000004,"objects_trimmed":0,"snaptrim_duration":0.020407696999999999,"stat_sum":{"num_bytes":1343488,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0,3,5,7,1],"acting":[10,4,0,3,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"25","length":"1"},{"start":"28","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.12","version":"87'54","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.259010+0000","last_change":"2024-09-15T02:49:37.259447+0000","last_active":"2024-09-15T02:49:37.259010+0000","last_peered":"2024-09-15T02:49:37.259010+0000","last_clean":"2024-09-15T02:49:37.259010+0000","last_became_active":"2024-09-15T02:49:22.511352+0000","last_became_peered":"2024-09-15T02:49:22.511352+0000","last_unstale":"2024-09-15T02:49:37.259010+0000","last_undegraded":"2024-09-15T02:49:37.259010+0000","last_fullsized":"2024-09-15T02:49:37.259010+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'54","last_scrub_stamp":"2024-09-15T02:49:31.964027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.964027+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:34.561458+0000","scrub_duration":0.016273553999999999,"objects_trimmed":0,"snaptrim_duration":0.00062054099999999995,"stat_sum":{"num_bytes":8028160,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,5,9,12,2,7],"acting":[15,8,5,9,12,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.13","version":"86'89","reported_seq":134,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T02:49:37.257457+0000","last_change":"2024-09-15T02:49:37.259125+0000","last_active":"2024-09-15T02:49:37.257457+0000","last_peered":"2024-09-15T02:49:37.257457+0000","last_clean":"2024-09-15T02:49:37.257457+0000","last_became_active":"2024-09-15T02:49:18.472004+0000","last_became_peered":"2024-09-15T02:49:18.472004+0000","last_unstale":"2024-09-15T02:49:37.257457+0000","last_undegraded":"2024-09-15T02:49:37.257457+0000","last_fullsized":"2024-09-15T02:49:37.257457+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'89","last_scrub_stamp":"2024-09-15T02:49:31.119567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.119567+0000","objects_scrubbed":3,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.244504+0000","scrub_duration":0.61521208699999996,"objects_trimmed":0,"snaptrim_duration":0.0028525970000000001,"stat_sum":{"num_bytes":1130496,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,7,5,11,10],"acting":[8,0,3,7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"38","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":124657664,"num_objects":115,"num_object_clones":67,"num_object_copies":805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":8,"num_read":289,"num_read_kb":146379,"num_write":533,"num_write_kb":93128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":45481984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":226983936,"data_stored":311791616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1594,"ondisk_log_size":1594,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":22,"seq":94489280531,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322952,"kb_used_data":21428,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94048888,"statfs":{"total":96636764160,"available":96306061312,"internally_reserved":0,"allocated":21942272,"data_stored":28226021,"data_compressed":139508,"data_compressed_allocated":290816,"data_compressed_original":581632,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.212}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":8,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.4159999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.2629999999999999},{"interface":"front","average":{"1min":78.307000000000002,"5min":78.307000000000002,"15min":78.307000000000002},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":934.447,"5min":934.447,"15min":934.447},"last":1.1240000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":11,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":13.593999999999999,"5min":13.593999999999999,"15min":13.593999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":287.02499999999998,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.093}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.286},{"interface":"front","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.4730000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.304},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.341}]},{"osd":15,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":4.2679999999999998,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.4950000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280529,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321312,"kb_used_data":19788,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94050528,"statfs":{"total":96636764160,"available":96307740672,"internally_reserved":0,"allocated":20262912,"data_stored":28844517,"data_compressed":139508,"data_compressed_allocated":290816,"data_compressed_original":581632,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":50.710000000000001,"5min":50.710000000000001,"15min":50.710000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":743.34100000000001,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.78800000000000003}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":3.4929999999999999,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.151}]},{"osd":5,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":4.141,"5min":4.141,"15min":4.141},"last":1.046}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.1160000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":81.322000000000003,"5min":81.322000000000003,"15min":81.322000000000003},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":936.86800000000005,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.5349999999999999},{"interface":"front","average":{"1min":32.043999999999997,"5min":32.043999999999997,"15min":32.043999999999997},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":586.00900000000001,"5min":586.00900000000001,"15min":586.00900000000001},"last":0.92000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":1.1879999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":15,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.5580000000000001}]}]},{"osd":5,"up_from":21,"seq":90194313234,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310604,"kb_used_data":13240,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94061236,"statfs":{"total":96636764160,"available":96318705664,"internally_reserved":0,"allocated":13557760,"data_stored":17753916,"data_compressed":137482,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":4,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.101}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.113}]},{"osd":9,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":3.1160000000000001,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":0.82399999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":11,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.1399999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"max":{"1min":3.363,"5min":3.363,"15min":3.363},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":1.073}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280529,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318708,"kb_used_data":17188,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94053132,"statfs":{"total":96636764160,"available":96310407168,"internally_reserved":0,"allocated":17600512,"data_stored":24524604,"data_compressed":137482,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,5,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0609999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":0.95099999999999996},{"interface":"front","average":{"1min":43.116999999999997,"5min":43.116999999999997,"15min":43.116999999999997},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":503.59300000000002,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.4259999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":10.843,"5min":10.843,"15min":10.843},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":223.92500000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.5289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.774},{"interface":"front","average":{"1min":42.957000000000001,"5min":42.957000000000001,"15min":42.957000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":954.81700000000001,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.865}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4910000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.653},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.1099999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.048}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.748}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1829999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3159999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0219999999999998}]},{"osd":15,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":22.420999999999999,"5min":22.420999999999999,"15min":22.420999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":504.36000000000001,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.96},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.587},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":3.5609999999999999,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.9379999999999999}]}]},{"osd":13,"up_from":22,"seq":94489280529,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316040,"kb_used_data":14520,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94055800,"statfs":{"total":96636764160,"available":96313139200,"internally_reserved":0,"allocated":14868480,"data_stored":20339070,"data_compressed":137482,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.5740000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.167},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":551}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.321},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.6850000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.77}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.425},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":1.032,"5min":1.032,"15min":1.032},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.6399999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":2.0640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.754}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.948}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"min":{"1min":1.417,"5min":1.417,"15min":1.417},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.9930000000000001},{"interface":"front","average":{"1min":2,"5min":2,"15min":2},"min":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":841.86500000000001}]},{"osd":14,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.899,"15min":1.899},"min":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.831},{"interface":"front","average":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":3.069,"5min":3.069,"15min":3.069},"last":1.875}]},{"osd":15,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":38.848999999999997,"5min":38.848999999999997,"15min":38.848999999999997},"min":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"max":{"1min":663.61500000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":2.4209999999999998},{"interface":"front","average":{"1min":1.861,"5min":1.861,"15min":1.861},"min":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":2.0190000000000001}]}]},{"osd":1,"up_from":22,"seq":94489280529,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317636,"kb_used_data":16116,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94054204,"statfs":{"total":96636764160,"available":96311504896,"internally_reserved":0,"allocated":16502784,"data_stored":22993278,"data_compressed":137482,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":12.202,"5min":12.202,"15min":12.202},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":0.84399999999999997}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":0.876},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":1.018}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.048},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":4.3129999999999997,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.089},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":4.0119999999999996,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":1.2749999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.147,"5min":1.147,"15min":1.147},"max":{"1min":4.4809999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.256},{"interface":"front","average":{"1min":1.77,"5min":1.77,"15min":1.77},"min":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"max":{"1min":4.1299999999999999,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.3089999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":4.4119999999999999,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"max":{"1min":3.839,"5min":3.839,"15min":3.839},"last":1.345}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280529,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312336,"kb_used_data":14976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94059504,"statfs":{"total":96636764160,"available":96316932096,"internally_reserved":0,"allocated":15335424,"data_stored":20273534,"data_compressed":137482,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":472.28100000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":1.169}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":1.2190000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.3599999999999999,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":11.234,"5min":11.234,"15min":11.234},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":183.30199999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.087}]},{"osd":4,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":38.918999999999997,"5min":38.918999999999997,"15min":38.918999999999997},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":681.39599999999996,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.2829999999999999},{"interface":"front","average":{"1min":52.350000000000001,"5min":52.350000000000001,"15min":52.350000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":921.02099999999996,"5min":921.02099999999996,"15min":921.02099999999996},"last":1.554}]},{"osd":5,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1930000000000001,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":1.482}]},{"osd":9,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.512},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.677}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":11,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":3.133,"5min":3.133,"15min":3.133},"last":1.925},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.889}]},{"osd":13,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.776},{"interface":"front","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":3.0659999999999998,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.853}]},{"osd":15,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":3.367,"5min":3.367,"15min":3.367},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":34.311999999999998,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.7250000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317920,"kb_used_data":16400,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94053920,"statfs":{"total":96636764160,"available":96311214080,"internally_reserved":0,"allocated":16793600,"data_stored":24165278,"data_compressed":145087,"data_compressed_allocated":319488,"data_compressed_original":1085440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":1,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":46.866999999999997,"5min":46.866999999999997,"15min":46.866999999999997},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":917.20299999999997,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.74099999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":1.0649999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.95099999999999996}]},{"osd":5,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.80400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.103},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":1.01}]},{"osd":9,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.5270000000000001,"5min":2.5270000000000001,"15min":2.5270000000000001},"last":1.194},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.173}]},{"osd":11,"last update":"Sun Sep 15 02:49:29 2024","interfaces":[{"interface":"back","average":{"1min":28.021999999999998,"5min":28.021999999999998,"15min":28.021999999999998},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":531.54300000000001,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.091},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":1.218}]},{"osd":13,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.115},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3570000000000002,"5min":3.3570000000000002,"15min":3.3570000000000002},"last":1.2390000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280529,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311400,"kb_used_data":14040,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94060440,"statfs":{"total":96636764160,"available":96317890560,"internally_reserved":0,"allocated":14376960,"data_stored":19478910,"data_compressed":137482,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.23}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.52},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.391}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.474}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7769999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":1.825},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":1.7070000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4160000000000004,"5min":6.4160000000000004,"15min":6.4160000000000004},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":121.304,"5min":121.304,"15min":121.304},"last":1.948},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.752}]},{"osd":13,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5979999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.8859999999999999},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.9079999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":1.26,"5min":1.26,"15min":1.26},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.804},{"interface":"front","average":{"1min":18.201000000000001,"5min":18.201000000000001,"15min":18.201000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":383.084,"5min":383.084,"15min":383.084},"last":1.9870000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311508,"kb_used_data":14144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94060332,"statfs":{"total":96636764160,"available":96317779968,"internally_reserved":0,"allocated":14483456,"data_stored":18936293,"data_compressed":139508,"data_compressed_allocated":290816,"data_compressed_original":581632,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,4,5,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.752},{"interface":"front","average":{"1min":42.505000000000003,"5min":42.505000000000003,"15min":42.505000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":913.93899999999996,"5min":913.93899999999996,"15min":913.93899999999996},"last":1.0820000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.167}]},{"osd":5,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.206},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.88}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.998},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":1.024,"5min":1.024,"15min":1.024},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.25}]},{"osd":13,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.117,"5min":1.117,"15min":1.117},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":1.23},{"interface":"front","average":{"1min":5.4930000000000003,"5min":5.4930000000000003,"15min":5.4930000000000003},"min":{"1min":1.153,"5min":1.153,"15min":1.153},"max":{"1min":92.698999999999998,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.0269999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308320,"kb_used_data":10956,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94063520,"statfs":{"total":96636764160,"available":96321044480,"internally_reserved":0,"allocated":11218944,"data_stored":14119416,"data_compressed":139508,"data_compressed_allocated":290816,"data_compressed_original":581632,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":7,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":10.6,"5min":10.6,"15min":10.6},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":182.642,"5min":182.642,"15min":182.642},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.1379999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.1799999999999999},{"interface":"front","average":{"1min":43.805,"5min":43.805,"15min":43.805},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":849.11599999999999,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.605}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.278},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.448}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":39.085999999999999,"5min":39.085999999999999,"15min":39.085999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":423.70800000000003,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.198}]},{"osd":14,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.657},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":1.103,"5min":1.103,"15min":1.103},"max":{"1min":3.012,"5min":3.012,"15min":3.012},"last":1.3839999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.508},{"interface":"front","average":{"1min":33.267000000000003,"5min":33.267000000000003,"15min":33.267000000000003},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":603.77599999999995,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.7330000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280529,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304916,"kb_used_data":7552,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94066924,"statfs":{"total":96636764160,"available":96324530176,"internally_reserved":0,"allocated":7733248,"data_stored":8507318,"data_compressed":139508,"data_compressed_allocated":290816,"data_compressed_original":581632,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.86499999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":215.09100000000001},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.004},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.089}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.82899999999999996}]},{"osd":12,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.361},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.0429999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":1.002,"5min":1.002,"15min":1.002},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.347,"5min":2.347,"15min":2.347},"last":1.306}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626445,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307484,"kb_used_data":9616,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94064356,"statfs":{"total":96636764160,"available":96321900544,"internally_reserved":0,"allocated":9846784,"data_stored":11938652,"data_compressed":145595,"data_compressed_allocated":303104,"data_compressed_original":606208,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8320000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1459999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0680000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3740000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280530,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322780,"kb_used_data":21260,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94049060,"statfs":{"total":96636764160,"available":96306237440,"internally_reserved":0,"allocated":21770240,"data_stored":29895582,"data_compressed":145087,"data_compressed_allocated":319488,"data_compressed_original":1085440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.0009999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.099}]},{"osd":2,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.496,"5min":2.496,"15min":2.496},"last":1.127},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":4.1559999999999997,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.409}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.343},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.0580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.4470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.72},{"interface":"front","average":{"1min":31.167999999999999,"5min":31.167999999999999,"15min":31.167999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":713.26999999999998,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.472}]},{"osd":13,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":28.263000000000002,"5min":28.263000000000002,"15min":28.263000000000002},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":637.95699999999999,"5min":637.95699999999999,"15min":637.95699999999999},"last":1.675},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.778}]},{"osd":14,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.742},{"interface":"front","average":{"1min":1.976,"5min":1.976,"15min":1.976},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":5.0839999999999996,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.532}]},{"osd":15,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":52.142000000000003,"5min":52.142000000000003,"15min":52.142000000000003},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":586.40499999999997,"5min":586.40499999999997,"15min":586.40499999999997},"last":1.804},{"interface":"front","average":{"1min":1.631,"5min":1.631,"15min":1.631},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.8559999999999999,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.6579999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280529,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":10256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":10502144,"data_stored":13304293,"data_compressed":139508,"data_compressed_allocated":290816,"data_compressed_original":581632,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.0920000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.6699999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":0.99299999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.115},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.2410000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.131}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":3.7440000000000002,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.22},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.867}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.5389999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":2.0920000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.724}]},{"osd":14,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":3.819,"5min":3.819,"15min":3.819},"last":1.5129999999999999},{"interface":"front","average":{"1min":15.798999999999999,"5min":15.798999999999999,"15min":15.798999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":285.95499999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.992}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17698816,"data_stored":24899584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16596992,"data_stored":23867392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15683584,"data_stored":22859776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13414400,"data_stored":18149376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19193856,"data_stored":28057600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":16977920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8888320,"data_stored":11423744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20873216,"data_stored":27439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10145792,"data_stored":13332480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":7815168,"data_stored":8871936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9211904,"data_stored":11591680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20660224,"data_stored":29089792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":10584064,"data_stored":13668352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":14962688,"data_stored":21213184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":15429632,"data_stored":21147648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13320192,"data_stored":19202048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:49:43.029 INFO:tasks.ceph.ceph_manager.ceph:pg 3.19 in trimming, state: active+clean+snaptrim 2024-09-15T02:49:43.029 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting 2024-09-15T02:49:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:43.029 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:43.011+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:43.119 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:43.116+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:43.137 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:43.137 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:43.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:43.202 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:43.220 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:43.212+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:43.232 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:43.249 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:43.250 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:43.250 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:43.250 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:43.250 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:43.260 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:43.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:43.263+0000 7fedcffff700 0 -- 172.21.15.6:0/985100625 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7fedd01933b0 msgr2=0x7fedd01957b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:43.298 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:43.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:49:43.320 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:43.316+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:43.331 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:43.334 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:43.334 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:43.334 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:43.334 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:43.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=20, reqid=client.4526.0:249, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:43.360 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.764837763000003, 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020019123999999999, 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592407+0000", 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9999999999999997e-07 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.592951+0000", 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054339899999999997 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593028+0000", 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6892999999999997e-05 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.593044+0000", 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6407999999999999e-05 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595661+0000", 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002617032 2024-09-15T02:49:43.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.595686+0000", 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5358999999999999e-05 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612387+0000", 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016700194000000002 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612426+0000", 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9138999999999999e-05 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 41/33 ECSubWrite(tid=21, reqid=client.4526.0:250, at_version=41'14, trim_to=0'0, roll_forward_to=39'11))", 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.752925011999999, 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086779219999999994, 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.365 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604319+0000", 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604321+0000", 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.902e-06 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604517+0000", 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019603099999999999 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604572+0000", 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4144000000000002e-05 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.604581+0000", 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4360000000000008e-06 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608062+0000", 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034810710000000001 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.608079+0000", 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7059999999999999e-05 2024-09-15T02:49:43.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612964+0000", 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004885197 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:45.612997+0000", 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3080999999999997e-05 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.369 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/33 ECSubWrite(tid=30, reqid=osd.8.0:30, at_version=44'23, trim_to=0'0, roll_forward_to=43'19))", 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 54.757278337000002, 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080113409999999999, 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599966+0000", 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.599967+0000", 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3120000000000001e-06 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600013+0000", 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126999999999997e-05 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600081+0000", 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8295999999999998e-05 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600094+0000", 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3522e-05 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600358+0000", 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026319700000000001 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.600388+0000", 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029e-05 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607948+0000", 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075605949999999998 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:48.607977+0000", 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9263000000000001e-05 2024-09-15T02:49:43.373 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.086353701, 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012340179999999999, 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.270891+0000", 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0999999999999999e-07 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271155+0000", 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026426800000000001 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271179+0000", 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3893999999999999e-05 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271190+0000", 2024-09-15T02:49:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0525e-05 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271256+0000", 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6328999999999995e-05 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.271268+0000", 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1831000000000001e-05 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283197+0000", 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011929294999999999 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283231+0000", 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3627999999999997e-05 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.18s1 77/76 ECSubWrite(tid=10, reqid=client.4526.0:769, at_version=77'48, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:43.378 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.868090886000001, 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017491599, 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:43.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489154+0000", 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489188+0000", 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3583e-05 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489252+0000", 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3886000000000005e-05 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.489261+0000", 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6680000000000007e-06 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.492869+0000", 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036079990000000002 2024-09-15T02:49:43.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.492877+0000", 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5699999999999993e-06 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506625+0000", 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013747786 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506645+0000", 2024-09-15T02:49:43.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0304000000000001e-05 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.18s1 77/76 [PushOp(3:1c51ca92:::smithi19125844-38:26, version: 74'43, data_included: [0~106496], data_size: 106496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:1c51ca92:::smithi19125844-38:26@74'43, size: 425984, copy_subset: [], clone_subset: {}, snapset: 27=[]:{26=[26,25,24,21,1d,18,b],27=[27]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:425984, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.491500+0000", 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.865744278000001, 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015075897, 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491500+0000", 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491500+0000", 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491501+0000", 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6000000000000002e-07 2024-09-15T02:49:43.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491651+0000", 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015037699999999999 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491694+0000", 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2834999999999999e-05 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.491704+0000", 2024-09-15T02:49:43.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3740000000000004e-06 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493618+0000", 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019141970000000001 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506576+0000", 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012958254000000001 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.18s1 77/76 [PushOp(3:1a9d0ab8:::smithi19125844-25:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:1a9d0ab8:::smithi19125844-25:head@72'42, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.493005+0000", 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.86423988, 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013545830999999999, 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493005+0000", 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493005+0000", 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493006+0000", 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.017e-06 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493359+0000", 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035336500000000002 2024-09-15T02:49:43.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493406+0000", 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7222999999999998e-05 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.493418+0000", 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.163e-05 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.496752+0000", 2024-09-15T02:49:43.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033343029999999998 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.506550+0000", 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097982929999999996 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.859403642, 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019140009999999999, 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.497841+0000", 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6e-07 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499086+0000", 2024-09-15T02:49:43.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012450510000000001 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499145+0000", 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8804999999999997e-05 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499154+0000", 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9239999999999996e-06 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499307+0000", 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015314899999999999 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.499329+0000", 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2248000000000001e-05 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.516950+0000", 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017620523999999999 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.516981+0000", 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1149000000000003e-05 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.12s3 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.522939+0000", 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.834305126, 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058133600000000001, 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.522939+0000", 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.522939+0000", 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.522940+0000", 2024-09-15T02:49:43.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1260000000000001e-06 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523514+0000", 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057330300000000005 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523569+0000", 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5475e-05 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523581+0000", 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2225999999999999e-05 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523632+0000", 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0643000000000003e-05 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.523644+0000", 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2092999999999999e-05 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.528730+0000", 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050854580000000002 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.528753+0000", 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3036000000000001e-05 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.12s3 78/77 ECSubWrite(tid=130, reqid=client.4526.0:744, at_version=78'42, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:43.399 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.534465+0000", 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.822779316999998, 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071005670000000003, 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534465+0000", 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534465+0000", 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534466+0000", 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.534989+0000", 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052318000000000004 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535051+0000", 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2113999999999995e-05 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535073+0000", 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1359e-05 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535149+0000", 2024-09-15T02:49:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5894000000000002e-05 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.535168+0000", 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8838000000000002e-05 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.541535+0000", 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063670369999999999 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.541566+0000", 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1164000000000002e-05 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.12s3 78/77 ECSubRead(tid=134, to_read={3:48b768c1:::smithi19125844-48:head=0,4096,0}, subchunks={3:48b768c1:::smithi19125844-48:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.561746+0000", 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.795498551000001, 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090836619999999993, 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561746+0000", 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561746+0000", 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561747+0000", 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.561757+0000", 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0322999999999999e-05 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570342+0000", 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085848009999999995 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570355+0000", 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3115000000000001e-05 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570469+0000", 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113448 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570830+0000", 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036096199999999999 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.407 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 23, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.569080+0000", 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.788164688999998, 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19609521199999999, 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569080+0000", 2024-09-15T02:49:43.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569080+0000", 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.569081+0000", 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.46e-06 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.570304+0000", 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001222919 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.748181+0000", 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.17787734099999999 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.748210+0000", 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8915999999999999e-05 2024-09-15T02:49:43.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.748415+0000", 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020410200000000001 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.765175+0000", 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016760474000000001 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.773808+0000", 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.583436599999999, 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097636119999999996, 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773808+0000", 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773808+0000", 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.773810+0000", 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8169999999999999e-06 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774581+0000", 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077112000000000003 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774666+0000", 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5634e-05 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774690+0000", 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3315999999999998e-05 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.774786+0000", 2024-09-15T02:49:43.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6129000000000001e-05 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.783571+0000", 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087855959999999997 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:b, version: 72'38, data_included: [524288~172032], data_size: 172032, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:b@72'38, size: 2785280, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2785280, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.798905+0000", 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.558339179000001, 2024-09-15T02:49:43.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092489289999999995, 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.798905+0000", 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.798905+0000", 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.798907+0000", 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799185+0000", 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027742700000000001 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799269+0000", 2024-09-15T02:49:43.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4704999999999996e-05 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799288+0000", 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8389e-05 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.799387+0000", 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9470999999999997e-05 2024-09-15T02:49:43.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.808154+0000", 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087671139999999995 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 43, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.828736+0000", 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.528508801000001, 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016867099999999999, 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.828736+0000", 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.828736+0000", 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.828737+0000", 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7770000000000001e-06 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.829950+0000", 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012128830000000001 2024-09-15T02:49:43.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.830039+0000", 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8721000000000007e-05 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.830063+0000", 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4193e-05 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.830199+0000", 2024-09-15T02:49:43.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013589 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.845603+0000", 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015403636 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.850192+0000", 2024-09-15T02:49:43.423 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.507052592000001, 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006788924, 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850192+0000", 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850192+0000", 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850193+0000", 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6700000000000002e-07 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850699+0000", 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050657600000000001 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850740+0000", 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0948000000000002e-05 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850749+0000", 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4370000000000007e-06 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.850805+0000", 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6122000000000002e-05 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.856981+0000", 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061758739999999996 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.861481+0000", 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.495763673999999, 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074314619999999998, 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.427 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861481+0000", 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861481+0000", 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861482+0000", 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.083e-06 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861799+0000", 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000317368 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861844+0000", 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5117000000000002e-05 2024-09-15T02:49:43.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861853+0000", 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.861899+0000", 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6385999999999999e-05 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.868912+0000", 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070131029999999997 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPush(3.12s3 78/77 [PushOp(3:48b768c1:::smithi19125844-48:18, version: 70'36, data_included: [786432~204800], data_size: 204800, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48b768c1:::smithi19125844-48:18@70'36, size: 3964928, copy_subset: [], clone_subset: {}, snapset: 27=[]:{b=[b],18=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3964928, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:22.873803+0000", 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.483441272, 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062309849999999996, 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.873803+0000", 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.873803+0000", 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.873804+0000", 2024-09-15T02:49:43.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.08e-06 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874311+0000", 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050656599999999996 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874353+0000", 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2154999999999999e-05 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874361+0000", 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2779999999999993e-06 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.874410+0000", 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8786999999999999e-05 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.880034+0000", 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056241190000000003 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.434 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.cs6 93/76 ECSubWrite(tid=156, reqid=client.4526.0:952, at_version=93'25, trim_to=0'0, roll_forward_to=85'23))", 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.0715285229999996, 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038285843999999999, 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:43.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.285716+0000", 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4900000000000001e-07 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289183+0000", 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034662299999999998 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289249+0000", 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6700999999999997e-05 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289261+0000", 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1240000000000001e-05 2024-09-15T02:49:43.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289437+0000", 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017670500000000001 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.289466+0000", 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8433999999999999e-05 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.323978+0000", 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034512481999999997 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:37.324002+0000", 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3603000000000001e-05 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 95/73 ECSubWriteReply(tid=23, last_complete=95'65, committed=1, applied=1))", 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:39.253286+0000", 2024-09-15T02:49:43.439 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.1039585880000002, 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.15128029300000001, 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.253286+0000", 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.253286+0000", 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.253287+0000", 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.15e-06 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.253296+0000", 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7749999999999998e-06 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.403469+0000", 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.150172742 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.403484+0000", 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.594e-05 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.403613+0000", 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012839700000000001 2024-09-15T02:49:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:39.404566+0000", 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00095328899999999996 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.443 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:43.445 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:43.439+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:43.545 INFO:tasks.ceph.osd.3.smithi191.stderr:2024-09-15T02:49:43.544+0000 7f2361827700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22448) UID: 0 2024-09-15T02:49:43.550 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:43.551 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:43.551 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-15T02:49:43.551 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:43.647 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:43.644+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:43.660 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:43.660 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-15T02:49:43.660 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-15T02:49:43.660 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-15T02:49:43.660 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:43.748 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:43.744+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:388 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.016428826999999, 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.205927366, 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 388 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752805+0000", 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8480000000000003e-06 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752820+0000", 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752827+0000", 2024-09-15T02:49:43.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8059999999999997e-06 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770449+0000", 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017621527000000001 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770567+0000", 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118576 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776872+0000", 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063044340000000003 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776986+0000", 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011429699999999999 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958728+0000", 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.181741654 2024-09-15T02:49:43.777 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:389 3.7s0 3:e10d9d4d:::smithi19125844-8:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.016393565999998, 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47945425899999999, 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.778 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 389 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752836+0000", 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752842+0000", 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8579999999999996e-06 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752860+0000", 2024-09-15T02:49:43.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7816e-05 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752867+0000", 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0260000000000002e-06 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770793+0000", 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017925963999999999 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770931+0000", 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137978 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955584+0000", 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.184652765 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.959571+0000", 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003987488 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232070+0000", 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.272499243 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232290+0000", 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021937500000000001 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:390 3.7s0 3:e10d9d4d:::smithi19125844-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:43.783 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.016355376, 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20558775300000001, 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 390 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:43.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8199999999999999e-07 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752878+0000", 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4489999999999999e-06 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752888+0000", 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0122000000000001e-05 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752894+0000", 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9220000000000002e-06 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771143+0000", 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018248665000000001 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771198+0000", 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5158000000000003e-05 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958258+0000", 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18706018699999999 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958359+0000", 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101025 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958461+0000", 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102543 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.788 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:402 3.1s0 3:801a1da7:::smithi19125844-39:e [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.010977613000001, 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52960550699999998, 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 402 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758252+0000", 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5000000000000002e-07 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758257+0000", 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3800000000000004e-06 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758267+0000", 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0455000000000001e-05 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758604+0000", 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033705300000000003 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761065+0000", 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024613840000000001 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761208+0000", 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014282499999999999 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287857+0000", 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52664865999999999 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 49/33 ECSubReadReply(tid=183, attrs_read=0))", 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.006005113000001, 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52471420199999996, 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763572+0000", 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034791800000000001 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286378+0000", 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52280558799999999 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286384+0000", 2024-09-15T02:49:43.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286558+0000", 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017370499999999999 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287938+0000", 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013805550000000001 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:433 3.7s0 3:ec23d144:::smithi19125844-41:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.486235739999998, 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27191602199999998, 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-15T02:49:43.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282994+0000", 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1790000000000001e-06 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283000+0000", 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0479999999999998e-06 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283008+0000", 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1440000000000003e-06 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283012+0000", 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7129999999999996e-06 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298201+0000", 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015188495 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298355+0000", 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015441200000000001 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298951+0000", 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059618300000000002 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300677+0000", 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001725285 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554804+0000", 2024-09-15T02:49:43.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25412710900000002 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554909+0000", 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105454 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:434 3.7s0 3:ec23d144:::smithi19125844-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:43.802 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.486211816999997, 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27651220700000001, 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 434 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283018+0000", 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0500000000000003e-07 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283021+0000", 2024-09-15T02:49:43.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.959e-06 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283028+0000", 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0169999999999999e-06 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283032+0000", 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0509999999999998e-06 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299373+0000", 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016340659 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299481+0000", 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108218 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554991+0000", 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25551037700000001 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.555256+0000", 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026495400000000002 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559529+0000", 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042733670000000001 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:437 3.6s0 3:6e9f7d10:::smithi19125844-33:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:43.807 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.486144879999998, 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20344801100000001, 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 437 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283085+0000", 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283088+0000", 2024-09-15T02:49:43.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.993e-06 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283094+0000", 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5810000000000002e-06 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6289999999999999e-06 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284035+0000", 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093763400000000002 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284100+0000", 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4882999999999995e-05 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486532+0000", 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20243185599999999 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:438 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [delete] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e50)", 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.486111071000003, 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27354936000000002, 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 438 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8100000000000003e-07 2024-09-15T02:49:43.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283121+0000", 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8830000000000002e-06 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283129+0000", 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3390000000000004e-06 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283133+0000", 2024-09-15T02:49:43.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299523+0000", 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389989000000001 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299625+0000", 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010208 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300116+0000", 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049149000000000003 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300972+0000", 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085554200000000004 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556582+0000", 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25561025399999998 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556667+0000", 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5208000000000001e-05 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.6s0 50/33 ECSubReadReply(tid=200, attrs_read=0))", 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.473941478, 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19127902199999999, 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.817 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2600000000000001e-07 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295338+0000", 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9951999999999999e-05 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485520+0000", 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19018270600000001 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485530+0000", 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0820000000000005e-06 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485655+0000", 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012580999999999999 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486567+0000", 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000911246 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=203, last_complete=50'62, committed=1, applied=1))", 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.465896295999997, 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25159346799999999, 2024-09-15T02:49:43.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303336+0000", 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9550000000000001e-06 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554637+0000", 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25130088299999997 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554647+0000", 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9329999999999996e-06 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554704+0000", 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6514000000000001e-05 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554926+0000", 2024-09-15T02:49:43.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222647 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 50/33 ECSubWriteReply(tid=205, last_complete=50'35, committed=1, applied=1))", 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.449888866000002, 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.23186345, 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319341+0000", 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6700000000000003e-07 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319347+0000", 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6879999999999996e-06 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550925+0000", 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.231578591 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550946+0000", 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0903e-05 2024-09-15T02:49:43.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551110+0000", 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016408300000000001 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551204+0000", 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3418000000000006e-05 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=208, last_complete=50'64, committed=1, applied=1))", 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.210031700000002, 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.60425413100000003, 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559198+0000", 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9500000000000001e-07 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559203+0000", 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.13e-06 2024-09-15T02:49:43.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163222+0000", 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.604018638 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163237+0000", 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4622e-05 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163384+0000", 2024-09-15T02:49:43.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014714099999999999 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163451+0000", 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7805000000000006e-05 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 54/53 ECSubWrite(tid=87, reqid=client.4526.0:461, at_version=0'0, trim_to=0'0, roll_forward_to=54'39))", 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:43.832 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.848129032999999, 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.41018363000000002, 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:43.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921102+0000", 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921127+0000", 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4987e-05 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329820+0000", 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.40869367400000001 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329833+0000", 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2669e-05 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329958+0000", 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012490099999999999 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329981+0000", 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2463000000000002e-05 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331170+0000", 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001189701 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331284+0000", 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113435 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.836 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPushReply(3.1s0 54/53 [PushReplyOp(3:801a1da7:::smithi19125844-39:head)])", 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.47441225, 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99244340499999995, 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:43.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294818+0000", 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294837+0000", 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9052999999999999e-05 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286615+0000", 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99177708899999995 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286629+0000", 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4583000000000001e-05 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286721+0000", 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1640999999999999e-05 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.287260+0000", 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053939199999999997 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.840 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 55/53 ECSubRead(tid=88, to_read={3:94f52c52:::smithi19125844-20:head=0,110592,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.439341777000003, 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59246917700000001, 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329888+0000", 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329900+0000", 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.188e-05 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921816+0000", 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59191607099999999 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921831+0000", 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4765e-05 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922011+0000", 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018023900000000001 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922356+0000", 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034564200000000001 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 62/53 ECSubWriteReply(tid=316, last_complete=62'23, committed=1, applied=1))", 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.411787279999999, 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58891104599999999, 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.844 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6400000000000002e-07 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357447+0000", 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9529999999999998e-06 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945262+0000", 2024-09-15T02:49:43.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58781430300000004 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945281+0000", 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9366000000000001e-05 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945429+0000", 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014825600000000001 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.946353+0000", 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092350400000000001 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.0.22:2 3.7s0 3:e10d9d4d:::smithi19125844-8:head [assert-version v25,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.293311589999998, 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.014934445, 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475920+0000", 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-09-15T02:49:43.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475928+0000", 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1750000000000005e-06 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475952+0000", 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4244e-05 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475960+0000", 2024-09-15T02:49:43.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0940000000000008e-06 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476051+0000", 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1032000000000005e-05 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476057+0000", 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3029999999999999e-06 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484706+0000", 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0086496460000001 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484954+0000", 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024785999999999997 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.490852+0000", 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058976979999999998 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:912 3.bs0 3:d6271403:::smithi19125844-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.453732123, 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18619464499999999, 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 912 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:43.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7899999999999998e-07 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315501+0000", 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2399999999999999e-06 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315519+0000", 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8094999999999999e-05 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315525+0000", 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8429999999999998e-06 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.340958+0000", 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025432834000000001 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.341123+0000", 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016572000000000001 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.499134+0000", 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.15801039 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.499395+0000", 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026174100000000001 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.501692+0000", 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022962030000000001 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.859 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 88/76 ECSubWriteReply(tid=531, last_complete=88'112, committed=1, applied=1))", 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.268483345, 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74370967399999999, 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:43.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7700000000000004e-07 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:43.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500750+0000", 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8229999999999997e-06 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244204+0000", 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74345404000000004 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244216+0000", 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1423000000000001e-05 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:43.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244348+0000", 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000132464 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244455+0000", 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107347 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:43.864 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:43.864 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:43.865 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:43.866 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:43.860+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:43.966 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:43.964+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:43.976 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:43.977 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:43.977 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:43.977 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:43.986 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:44.067 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:44.059+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:44.091 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:44.091 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:44.091 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:44.091 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:44.091 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:44.100 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:44.168 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:44.163+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 79.461829394000006, 2024-09-15T02:49:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:49:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:403 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~606208] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.441090699999997, 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.195887642, 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 403 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757022+0000", 2024-09-15T02:49:44.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.593e-06 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757800+0000", 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077824999999999997 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757830+0000", 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0057e-05 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757844+0000", 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3947999999999999e-05 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757925+0000", 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0249999999999999e-05 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.758082+0000", 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015777599999999999 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759386+0000", 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001303217 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777348+0000", 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017962568000000002 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952681+0000", 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17533264000000001 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952907+0000", 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000226343 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:404 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~720896] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.436872096000002, 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.192160255, 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.212 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 404 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:44.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761280+0000", 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2138999999999998e-05 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762400+0000", 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001119451 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762433+0000", 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3238000000000003e-05 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762446+0000", 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3308e-05 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768118+0000", 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056721360000000004 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768285+0000", 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000166482 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769662+0000", 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013768140000000001 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780475+0000", 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010813578000000001 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.216 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953275+0000", 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17280025199999999 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953398+0000", 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012285700000000001 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.217 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:405 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~49152] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.435641357999998, 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19129260000000001, 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 405 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.218 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762470+0000", 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4699999999999999e-06 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.219 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762527+0000", 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7151999999999998e-05 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762550+0000", 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2311999999999999e-05 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762559+0000", 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.220 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777614+0000", 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015055628999999999 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777696+0000", 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1840000000000002e-05 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.221 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.222 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.222 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778120+0000", 2024-09-15T02:49:44.222 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042372900000000002 2024-09-15T02:49:44.222 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.222 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.222 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780527+0000", 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00240712 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953622+0000", 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17309514500000001 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953761+0000", 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013926300000000001 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.223 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:406 3.es0 3:71c1a91c:::smithi19125844-45:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.435541397999998, 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19225968700000001, 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 406 2024-09-15T02:49:44.224 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.225 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762570+0000", 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.034e-06 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762575+0000", 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4079999999999997e-06 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762592+0000", 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.715e-05 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.226 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762600+0000", 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6480000000000003e-06 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779660+0000", 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017060035000000001 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779757+0000", 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6537000000000001e-05 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.227 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780137+0000", 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038070299999999998 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781414+0000", 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012764230000000001 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954649+0000", 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17323481900000001 2024-09-15T02:49:44.228 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954828+0000", 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017992999999999999 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:407 3.es0 3:71c1a91c:::smithi19125844-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.435501526000003, 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19532122900000001, 2024-09-15T02:49:44.229 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 407 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.230 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762610+0000", 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762614+0000", 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0659999999999997e-06 2024-09-15T02:49:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762626+0000", 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2490000000000001e-05 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762639+0000", 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2704000000000001e-05 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.232 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781806+0000", 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019166823999999999 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.782082+0000", 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027635000000000002 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955680+0000", 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17359843999999999 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.233 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955807+0000", 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012658299999999999 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.957930+0000", 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021228499999999999 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.13.22:5 3.es0 3:740f3d5b:::smithi19125844-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:44.234 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.422970546000002, 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.177222558, 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.235 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775141+0000", 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775146+0000", 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5630000000000002e-06 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.236 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775159+0000", 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3509999999999999e-05 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775165+0000", 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8130000000000001e-06 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778738+0000", 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003572867 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778948+0000", 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00021028299999999999 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952362+0000", 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17341393199999999 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:529 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~802816] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:44.238 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 39.943285089, 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.030230086, 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 529 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.239 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254827+0000", 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6390000000000001e-06 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256167+0000", 2024-09-15T02:49:44.240 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013402900000000001 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256196+0000", 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8762999999999999e-05 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256203+0000", 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6229999999999997e-06 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.241 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036663860000000002 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.260012+0000", 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014185099999999999 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017728349999999999 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.242 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277478+0000", 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015693650999999999 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285001+0000", 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075225120000000003 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285055+0000", 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4536000000000001e-05 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.243 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:530 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~638976] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 39.939363935000003, 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026488489, 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.244 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 530 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.245 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258748+0000", 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.66e-06 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259495+0000", 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00074681400000000005 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259522+0000", 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7552999999999999e-05 2024-09-15T02:49:44.246 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259538+0000", 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6220000000000001e-05 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269841+0000", 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010302477000000001 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.247 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269927+0000", 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5993000000000002e-05 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.270621+0000", 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00069455099999999996 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277769+0000", 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0071479769999999998 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.248 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285179+0000", 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074099320000000002 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285235+0000", 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5312000000000002e-05 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.249 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:533 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 39.938405240999998, 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027459596999999999, 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 533 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:44.250 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.89e-07 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.251 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259707+0000", 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5460000000000001e-06 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259711+0000", 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1439999999999996e-06 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259715+0000", 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3340000000000003e-06 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.252 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277889+0000", 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018173363000000001 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277948+0000", 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8913999999999999e-05 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.253 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286109+0000", 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0081616199999999996 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286156+0000", 2024-09-15T02:49:44.254 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6928000000000002e-05 2024-09-15T02:49:44.255 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.287165+0000", 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010083589999999999 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s2 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.664009110999999, 2024-09-15T02:49:44.256 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027643175999999998, 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.257 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534111+0000", 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8090000000000004e-06 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534425+0000", 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031419599999999998 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542230+0000", 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0078045069999999996 2024-09-15T02:49:44.258 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542239+0000", 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542375+0000", 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000135897 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542396+0000", 2024-09-15T02:49:44.259 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1237999999999999e-05 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561688+0000", 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019291689000000001 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561744+0000", 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6178999999999999e-05 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.260 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.8s2 77/76 ECSubRead(tid=258, to_read={3:173bfaf3:::smithi19125844-44:18=262144,262144,0}, subchunks={3:173bfaf3:::smithi19125844-44:18=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.622250352999998, 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119859681, 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:44.261 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575861+0000", 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0640000000000001e-06 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575876+0000", 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5165e-05 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694960+0000", 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119084367 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694980+0000", 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9930000000000001e-05 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.263 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695069+0000", 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8737000000000001e-05 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695719+0000", 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065041800000000005 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.264 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=112, attrs_read=0))", 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.617760319999999, 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13363014300000001, 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.265 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580351+0000", 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2419999999999999e-06 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580721+0000", 2024-09-15T02:49:44.266 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00037001299999999998 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712609+0000", 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13188792199999999 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712623+0000", 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.429e-05 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712782+0000", 2024-09-15T02:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000158791 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.713980+0000", 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011978850000000001 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.450733074999999, 2024-09-15T02:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21641586800000001, 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.269 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747379+0000", 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747806+0000", 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042687399999999999 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962958+0000", 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21515266799999999 2024-09-15T02:49:44.270 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962974+0000", 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6237999999999999e-05 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963059+0000", 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4968000000000004e-05 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963793+0000", 2024-09-15T02:49:44.271 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00073351999999999998 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.12s5 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 21.676268029999999, 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028389666000000001, 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.272 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521843+0000", 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.122e-06 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.273 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522442+0000", 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00059853399999999996 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533671+0000", 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011229075 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533686+0000", 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5265999999999999e-05 2024-09-15T02:49:44.274 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533798+0000", 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000112198 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533821+0000", 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2229999999999999e-05 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550165+0000", 2024-09-15T02:49:44.275 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016344826 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550232+0000", 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6414999999999993e-05 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:836 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~573440] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.599473734, 2024-09-15T02:49:44.276 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033215366000000003, 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 836 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598639+0000", 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1280000000000002e-06 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599731+0000", 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001092469 2024-09-15T02:49:44.278 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599759+0000", 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8472000000000001e-05 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599774+0000", 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4612e-05 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608764+0000", 2024-09-15T02:49:44.279 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089903399999999994 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608966+0000", 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020188699999999999 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.610666+0000", 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016996940000000001 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.280 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.630970+0000", 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020303799000000001 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631771+0000", 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00080119100000000003 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631852+0000", 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0773999999999996e-05 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.281 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:837 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~688128] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.594558254999999, 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037719309999999999, 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.282 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 837 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.283 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603553+0000", 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604429+0000", 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000875577 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604454+0000", 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4794999999999999e-05 2024-09-15T02:49:44.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604464+0000", 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9199999999999999e-06 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621603+0000", 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017138823000000001 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621666+0000", 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2996000000000005e-05 2024-09-15T02:49:44.285 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.622382+0000", 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071611400000000001 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631868+0000", 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0094864259999999992 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641117+0000", 2024-09-15T02:49:44.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0092490960000000001 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641271+0000", 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015396299999999999 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:838 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~294912] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.592616067000002, 2024-09-15T02:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.03629342, 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 838 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605496+0000", 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.415e-06 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605778+0000", 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028258100000000001 2024-09-15T02:49:44.289 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605797+0000", 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8817999999999998e-05 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605805+0000", 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5229999999999995e-06 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626744+0000", 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020938365 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626872+0000", 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012857999999999999 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.627645+0000", 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077218100000000004 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631967+0000", 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043226499999999999 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641643+0000", 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096753510000000004 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641788+0000", 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014495600000000001 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.292 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:839 3.es0 3:7773fe95:::smithi19125844-37:head [setxattr _header (57) in=7b] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.592295987, 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.036686074999999999, 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 839 2024-09-15T02:49:44.293 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.294 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605815+0000", 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1909999999999999e-06 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605822+0000", 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3369999999999996e-06 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605840+0000", 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7949e-05 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605848+0000", 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6980000000000004e-06 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631010+0000", 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025161701000000002 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631081+0000", 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0993000000000003e-05 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631379+0000", 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000298146 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.632106+0000", 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072663800000000005 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.297 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642331+0000", 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010225316 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642500+0000", 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000169106 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.298 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:840 3.es0 3:7773fe95:::smithi19125844-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 16.592252364, 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.039020892000000001, 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 840 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.299 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605859+0000", 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.02e-06 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.300 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605864+0000", 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1000000000000003e-06 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605882+0000", 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8128999999999999e-05 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605891+0000", 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5550000000000003e-06 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631895+0000", 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026004328 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631958+0000", 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2603999999999999e-05 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643304+0000", 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01134697 2024-09-15T02:49:44.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643450+0000", 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000145069 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.644879+0000", 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0014291169999999999 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.303 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.304 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:44.305 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:49:44.306 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:44.306 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:44.300+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:44.406 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:44.400+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:44.414 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:44.414 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:44.414 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:44.414 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:44.424 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:44.507 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:44.502+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:44.519 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:44.519 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:44.519 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:44.520 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:44.520 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:44.528 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:44.607 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:44.600+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:44.677 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.056185198999998, 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.678 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:49:44.679 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.680 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.681 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:416 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~638976] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.41397671, 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026381341999999999, 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.682 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 416 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.683 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261066+0000", 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1159999999999999e-06 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262194+0000", 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001127783 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262245+0000", 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0880000000000001e-05 2024-09-15T02:49:44.684 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262260+0000", 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5065999999999999e-05 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262382+0000", 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012151599999999999 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262403+0000", 2024-09-15T02:49:44.685 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1344000000000001e-05 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266706+0000", 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004303355 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266794+0000", 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7717000000000001e-05 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.686 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268260+0000", 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001466406 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285160+0000", 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016899773 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287372+0000", 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022116140000000002 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.687 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287445+0000", 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2772000000000001e-05 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:417 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.404582759999997, 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028569959999999998, 2024-09-15T02:49:44.688 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 417 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.689 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270459+0000", 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.035e-06 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271813+0000", 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001354037 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271834+0000", 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0302e-05 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271844+0000", 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9070000000000002e-06 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274939+0000", 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003095169 2024-09-15T02:49:44.691 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275041+0000", 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000102045 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276123+0000", 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010819880000000001 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.692 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285162+0000", 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009039432 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.298984+0000", 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013821599 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299027+0000", 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3445999999999997e-05 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.693 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:418 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~147456] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.403180259000003, 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027304571999999999, 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 418 2024-09-15T02:49:44.694 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.695 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271861+0000", 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1829999999999999e-06 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272037+0000", 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017563799999999999 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272050+0000", 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3506000000000001e-05 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.696 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272057+0000", 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999998e-06 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281725+0000", 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096680910000000002 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281887+0000", 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016169599999999999 2024-09-15T02:49:44.697 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282772+0000", 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088466100000000002 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285439+0000", 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026669340000000001 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299113+0000", 2024-09-15T02:49:44.698 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013674009000000001 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299164+0000", 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1573999999999998e-05 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:419 3.0s0 3:038350ba:::smithi19125844-11:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.402974962000002, 2024-09-15T02:49:44.699 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.031729234000000002, 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 419 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.700 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272066+0000", 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9000000000000005e-07 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272070+0000", 2024-09-15T02:49:44.701 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1740000000000002e-06 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272078+0000", 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5329999999999997e-06 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272083+0000", 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7859999999999996e-06 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.702 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284520+0000", 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012437190000000001 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284611+0000", 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0816000000000006e-05 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284926+0000", 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031568500000000001 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.703 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287991+0000", 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030649599999999998 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303704+0000", 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015713142999999999 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303794+0000", 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9957000000000006e-05 2024-09-15T02:49:44.704 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:420 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.402951954999999, 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033404771999999999, 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.705 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 420 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.706 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272089+0000", 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.51e-07 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272091+0000", 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1730000000000002e-06 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272096+0000", 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.904e-06 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.707 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272099+0000", 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2059999999999999e-06 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285286+0000", 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013187488000000001 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:44.708 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285374+0000", 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7392999999999995e-05 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304237+0000", 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018863347999999999 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304386+0000", 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014841 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.709 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.305493+0000", 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001107299 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:429 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.402878608000002, 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022788739999999998, 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.710 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 429 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:44.711 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272162+0000", 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272166+0000", 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9269999999999998e-06 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.712 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272174+0000", 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6569999999999997e-06 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272178+0000", 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6260000000000003e-06 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274127+0000", 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001949295 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.713 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274178+0000", 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0547999999999998e-05 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.294950+0000", 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020772112999999998 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 50/33 ECSubReadReply(tid=60, attrs_read=0))", 2024-09-15T02:49:44.714 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 50.397605339999998, 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017580023, 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.715 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0100000000000004e-07 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293753+0000", 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016317505999999999 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.716 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293859+0000", 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105707 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293862+0000", 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4819999999999999e-06 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293910+0000", 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7763000000000003e-05 2024-09-15T02:49:44.717 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.295015+0000", 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011048640000000001 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.12.22:2 3.9s0 3:94f52c52:::smithi19125844-20:head [assert-version v18,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 45.347183688999998, 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59485876100000001, 2024-09-15T02:49:44.718 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.719 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327859+0000", 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.446e-06 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327869+0000", 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7629999999999995e-06 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.720 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327898+0000", 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9054e-05 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327913+0000", 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4990999999999999e-05 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.721 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328005+0000", 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2015000000000006e-05 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328180+0000", 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017567599999999999 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.922715+0000", 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59453481600000002 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.722 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:630 3.9s0 3:9c49cf3d:::smithi19125844-14:head [copy-from ver 31] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.281129127, 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021438329999999998, 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.723 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 630 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.724 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393915+0000", 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7570000000000002e-06 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393951+0000", 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6347999999999998e-05 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.725 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393987+0000", 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6405999999999997e-05 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394003+0000", 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5956000000000001e-05 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394131+0000", 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012749300000000001 2024-09-15T02:49:44.726 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394155+0000", 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3754999999999999e-05 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401719+0000", 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075645340000000004 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.727 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401977+0000", 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000257543 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409969+0000", 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007991873 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414563+0000", 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045945830000000002 2024-09-15T02:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415199+0000", 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063606199999999996 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415349+0000", 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015002 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.729 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:631 3.9s0 3:9c49cf3d:::smithi19125844-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.280988188999999, 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017944699000000001, 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 631 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.730 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394053+0000", 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3179999999999999e-06 2024-09-15T02:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394059+0000", 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9120000000000003e-06 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394075+0000", 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5809999999999999e-05 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.732 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394085+0000", 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0586e-05 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394258+0000", 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017270999999999999 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394277+0000", 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9236999999999999e-05 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.733 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402333+0000", 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0080553299999999994 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402447+0000", 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00011408600000000001 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411771+0000", 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093238820000000004 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.734 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411860+0000", 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8857999999999995e-05 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411997+0000", 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013697000000000001 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.735 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 72/71 ECSubWriteReply(tid=134, last_complete=72'51, committed=1, applied=1))", 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.382198156000001, 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060209891000000001, 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.736 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292843+0000", 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3099999999999996e-07 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292848+0000", 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6729999999999997e-06 2024-09-15T02:49:44.737 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352898+0000", 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060049335000000002 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352910+0000", 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.256e-05 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.738 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352988+0000", 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7831999999999996e-05 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.353052+0000", 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3559999999999995e-05 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:688 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:44.739 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 26.236167142999999, 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037318727000000003, 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 688 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.740 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438875+0000", 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5809999999999999e-06 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.741 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438889+0000", 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3444999999999999e-05 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438927+0000", 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7842000000000001e-05 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438940+0000", 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3473e-05 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.742 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439086+0000", 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014553999999999999 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439109+0000", 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3283000000000001e-05 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448063+0000", 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089537490000000004 2024-09-15T02:49:44.743 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448274+0000", 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000210984 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476192+0000", 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027917830000000001 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.744 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:689 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 26.236078156000001, 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037724385999999999, 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 689 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.745 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438963+0000", 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5300000000000002e-07 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.746 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438968+0000", 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9180000000000002e-06 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438976+0000", 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6470000000000004e-06 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438981+0000", 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9130000000000002e-06 2024-09-15T02:49:44.747 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439306+0000", 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032486700000000001 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439319+0000", 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2442999999999999e-05 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.748 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448872+0000", 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095532159999999998 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449323+0000", 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00045103599999999998 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476686+0000", 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027363393 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.749 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:690 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 26.235996006000001, 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038145563, 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.750 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 690 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.751 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439045+0000", 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2200000000000003e-07 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439048+0000", 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.76e-06 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.752 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439064+0000", 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5336e-05 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439069+0000", 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8849999999999998e-06 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.753 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439382+0000", 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031364799999999998 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439397+0000", 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4661000000000001e-05 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449673+0000", 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010276298 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449816+0000", 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014270799999999999 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.477190+0000", 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027373544999999999 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.755 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDScrubReserve(3.19s4 RELEASE e75)", 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 24.393005116000001, 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98297010699999998, 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:44.756 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.757 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.757 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.757 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:44.757 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.757 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282036+0000", 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2270000000000001e-06 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282044+0000", 2024-09-15T02:49:44.758 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6720000000000004e-06 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264750+0000", 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98270659599999999 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264763+0000", 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2466000000000001e-05 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.759 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264912+0000", 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000149524 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264954+0000", 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1461000000000002e-05 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.265005+0000", 2024-09-15T02:49:44.760 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1161e-05 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s4 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.140727589000001, 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018239887999999999, 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.761 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534314+0000", 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7549999999999999e-06 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.762 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534957+0000", 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00064235200000000001 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535041+0000", 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4050999999999999e-05 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535056+0000", 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5588999999999999e-05 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.763 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535177+0000", 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120925 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535207+0000", 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9864000000000001e-05 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552499+0000", 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017291890000000001 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.764 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552552+0000", 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3461999999999997e-05 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.bs4 77/76 [PushOp(3:d394ce14:::smithi19125844-2:head, version: 71'77, data_included: [262144~241664], data_size: 241664, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d394ce14:::smithi19125844-2:head@71'77, size: 2015232, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2015232, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 23.038594175, 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.24636945099999999, 2024-09-15T02:49:44.765 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.766 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636447+0000", 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.637e-06 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636697+0000", 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024920600000000003 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867157+0000", 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.23046036 2024-09-15T02:49:44.767 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867176+0000", 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.929e-05 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867332+0000", 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015594099999999999 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.882815+0000", 2024-09-15T02:49:44.768 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015483017 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.177553613000001, 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025018460999999999, 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.769 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497487+0000", 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8000000000000002e-07 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.770 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499654+0000", 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00216738 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499711+0000", 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7398999999999997e-05 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499723+0000", 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2019e-05 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499830+0000", 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000106718 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499850+0000", 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9732e-05 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522437+0000", 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022587494 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.772 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522505+0000", 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7439000000000003e-05 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 93/76 ECSubWrite(tid=156, reqid=client.4526.0:952, at_version=93'25, trim_to=0'0, roll_forward_to=85'23))", 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 7.3889033819999996, 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038822470999999997, 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:44.773 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:44.774 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4300000000000003e-07 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292226+0000", 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060890609999999998 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292292+0000", 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6272000000000005e-05 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292303+0000", 2024-09-15T02:49:44.775 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0111e-05 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292409+0000", 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010671 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292423+0000", 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3424999999999999e-05 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:44.776 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.324890+0000", 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.032467754000000001 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.324959+0000", 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8595000000000003e-05 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:44.777 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:44.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:44.779 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:44.774+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:44.879 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:44.875+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:44.886 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:44.887 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:44.887 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:44.887 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:44.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:44.980 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:44.976+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:44.995 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:44.995 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:44.995 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:44.995 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:44.995 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:45.081 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:45.080+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 61.531312481000001, 2024-09-15T02:49:45.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:49:45.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.450365798999997, 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015141226000000001, 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650389+0000", 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650600+0000", 2024-09-15T02:49:45.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021155999999999999 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650679+0000", 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8641999999999997e-05 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650693+0000", 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4294000000000001e-05 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650801+0000", 2024-09-15T02:49:45.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108132 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650828+0000", 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6565999999999999e-05 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665503+0000", 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014675142 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665530+0000", 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6509999999999999e-05 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.408271565, 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011398508, 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.113 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.22e-07 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693204+0000", 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072156 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693258+0000", 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3134999999999999e-05 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693266+0000", 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5259999999999993e-06 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693333+0000", 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6580000000000003e-05 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693344+0000", 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0971999999999999e-05 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703854+0000", 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010510265 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703881+0000", 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7248000000000001e-05 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.846440936, 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016572159999999999, 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254314+0000", 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1100000000000002e-07 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255086+0000", 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077269100000000005 2024-09-15T02:49:45.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255147+0000", 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1005999999999998e-05 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255166+0000", 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8581999999999999e-05 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255267+0000", 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100973 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255289+0000", 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2565000000000002e-05 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270855+0000", 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015565733999999999 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270885+0000", 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0298e-05 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:45.122 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.831949981000001, 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018797873, 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8999999999999998e-07 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269160+0000", 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035577800000000001 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3349000000000001e-05 2024-09-15T02:49:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269211+0000", 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9439999999999999e-06 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269283+0000", 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2263999999999998e-05 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269296+0000", 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3145e-05 2024-09-15T02:49:45.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287519+0000", 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018222955999999998 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287602+0000", 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3146999999999995e-05 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.127 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(3.1s2 RELEASE e76)", 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.431119442, 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43008043499999998, 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:45.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669637+0000", 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.917e-06 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669648+0000", 2024-09-15T02:49:45.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1653e-05 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099516+0000", 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42986811200000002 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099534+0000", 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7997000000000001e-05 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099653+0000", 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011822100000000001 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099681+0000", 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8496999999999998e-05 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099715+0000", 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4038000000000002e-05 2024-09-15T02:49:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.617028458, 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012988501, 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483727+0000", 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2500000000000004e-07 2024-09-15T02:49:45.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483886+0000", 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000159028 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483939+0000", 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3640999999999997e-05 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483950+0000", 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0462000000000001e-05 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.484771+0000", 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082086099999999999 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.496714+0000", 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011943684 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.135 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:127fa005:::smithi19125844-18:head, version: 36'10, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:127fa005:::smithi19125844-18:head@36'10, size: 1212416, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.590960168999999, 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012919893999999999, 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509795+0000", 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8599999999999996e-07 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510077+0000", 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028238699999999999 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510142+0000", 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4518999999999997e-05 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510161+0000", 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8751999999999999e-05 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511930+0000", 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017696719999999999 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.522714+0000", 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010783578 2024-09-15T02:49:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.566185673, 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0218164, 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6199999999999996e-07 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534737+0000", 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016761400000000001 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534787+0000", 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9836000000000002e-05 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534795+0000", 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1550000000000007e-06 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534859+0000", 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4162e-05 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534875+0000", 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6305000000000001e-05 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556345+0000", 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021469637999999999 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556385+0000", 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9827999999999999e-05 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=251, reqid=client.4526.0:761, at_version=77'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.558390907, 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014142234, 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542364+0000", 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8700000000000005e-07 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542468+0000", 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010437099999999999 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542522+0000", 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3295999999999999e-05 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542529+0000", 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1709999999999996e-06 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544564+0000", 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00203476 2024-09-15T02:49:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544579+0000", 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5605000000000001e-05 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556487+0000", 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011907687 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556506+0000", 2024-09-15T02:49:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556999999999999e-05 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:18, version: 70'39, data_included: [262144~253952], data_size: 253952, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:18@70'39, size: 2064384, copy_subset: [], clone_subset: {}, snapset: 27=[]:{e=[b],18=[18],27=[27,26,25,24,21,1d]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2064384, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.402532019999999, 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010750387, 2024-09-15T02:49:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698224+0000", 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2570000000000002e-06 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698584+0000", 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035969000000000002 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698700+0000", 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116018 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698723+0000", 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3173e-05 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698830+0000", 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010714599999999999 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.708973+0000", 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010142103 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.154 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 81/76 ECSubRead(tid=114, to_read={3:517ce0e6:::smithi19125844-47:head=0,475136,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.611644512000002, 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69833002899999996, 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489111+0000", 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7269999999999999e-06 2024-09-15T02:49:45.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489134+0000", 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3033000000000001e-05 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187052+0000", 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69791777600000005 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187066+0000", 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3567000000000001e-05 2024-09-15T02:49:45.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187169+0000", 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010303599999999999 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187440+0000", 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027088999999999998 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "replica_scrub(pg: 3.0s3,from:0'0,to:85'34,epoch:86/71,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.683304101999999, 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98343883300000001, 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9999999999999999e-07 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417455+0000", 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-06 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400723+0000", 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98326817399999999 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400733+0000", 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5929999999999995e-06 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400839+0000", 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106413 2024-09-15T02:49:45.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400855+0000", 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5150000000000001e-05 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400889+0000", 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4353000000000001e-05 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=173, reqid=client.4526.0:901, at_version=88'59, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.819919441, 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019105225999999999, 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281129+0000", 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029418700000000001 2024-09-15T02:49:45.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281252+0000", 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123051 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281268+0000", 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5682999999999999e-05 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281408+0000", 2024-09-15T02:49:45.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013955900000000001 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281431+0000", 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2952e-05 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299857+0000", 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018425891 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299940+0000", 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3423000000000004e-05 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=175, reqid=client.4526.0:903, at_version=88'61, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.798628020000001, 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011439048, 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302127+0000", 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.119e-06 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302328+0000", 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000200834 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302386+0000", 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7895e-05 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302394+0000", 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0770000000000008e-06 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302465+0000", 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1017000000000001e-05 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302481+0000", 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5599000000000001e-05 2024-09-15T02:49:45.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313526+0000", 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011045743 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313565+0000", 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8763999999999997e-05 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=178, reqid=client.4526.0:905, at_version=0'0, trim_to=0'0, roll_forward_to=88'63))", 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.781012663, 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15873490300000001, 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319766+0000", 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3816e-05 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476980+0000", 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.157214144 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476993+0000", 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2286000000000001e-05 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477134+0000", 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141006 2024-09-15T02:49:45.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477167+0000", 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3847000000000003e-05 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478412+0000", 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012443930000000001 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478476+0000", 2024-09-15T02:49:45.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4579999999999995e-05 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.19s3 95/73 ECSubWrite(tid=23, reqid=osd.9.0:23, at_version=95'65, trim_to=0'0, roll_forward_to=93'63))", 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8492239670000004, 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01234181, 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.29e-07 2024-09-15T02:49:45.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251554+0000", 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4042000000000001e-05 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251590+0000", 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5329e-05 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251596+0000", 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7130000000000002e-06 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.256779+0000", 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051832880000000003 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.256788+0000", 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0890000000000006e-06 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263842+0000", 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070541850000000001 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263872+0000", 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9935000000000001e-05 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.181 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 95/71 ECSubWrite(tid=539, reqid=client.4526.0:958, at_version=95'82, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8303804189999999, 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025699248000000001, 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.73e-07 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270803+0000", 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042948800000000001 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270838+0000", 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4785e-05 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270851+0000", 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2347000000000001e-05 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270899+0000", 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8164999999999997e-05 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270909+0000", 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0614e-05 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.295820+0000", 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024910394999999998 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296073+0000", 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000253281 2024-09-15T02:49:45.185 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 95/71 ECSubWrite(tid=542, reqid=client.4526.0:959, at_version=95'83, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.278951+0000", 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8218035239999999, 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017151198999999999, 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.278951+0000", 2024-09-15T02:49:45.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.278951+0000", 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.278952+0000", 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1260000000000001e-06 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279680+0000", 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072793700000000001 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279725+0000", 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4894000000000001e-05 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279732+0000", 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-06 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.280070+0000", 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033776699999999999 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.280080+0000", 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0345e-05 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296083+0000", 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016002407999999999 2024-09-15T02:49:45.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296102+0000", 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9171999999999999e-05 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 95/71 ECSubWrite(tid=544, reqid=client.4526.0:960, at_version=95'84, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.286980+0000", 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.8137742210000001, 2024-09-15T02:49:45.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011104908, 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.286980+0000", 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.286980+0000", 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.286981+0000", 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.47e-07 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.287714+0000", 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00073319499999999996 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.287752+0000", 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8025000000000003e-05 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.287760+0000", 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7559999999999999e-06 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.289482+0000", 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017221350000000001 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.289492+0000", 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9690000000000006e-06 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.297996+0000", 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085040859999999992 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.298085+0000", 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8795000000000006e-05 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.194 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.195 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.195 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:45.197 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:45.194+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:45.297 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:45.294+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:45.303 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.303 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:45.303 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:45.303 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:45.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:45.398 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:45.394+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:45.412 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:45.413 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:45.498 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:45.491+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:45.508 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:45.522 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 61.949627006999997, 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.523 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:49:45.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 61.937110822000001, 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:49:45.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:49:45.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:49:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:49:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 61.936071662000003, 2024-09-15T02:49:45.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:49:45.534 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.534 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.534 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:45.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:49:45.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:49:45.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.868320656000002, 2024-09-15T02:49:45.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015031548, 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651048+0000", 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4900000000000001e-07 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651266+0000", 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000218233 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651324+0000", 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8544000000000001e-05 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651335+0000", 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336000000000001e-05 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651408+0000", 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3415000000000001e-05 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651424+0000", 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6001000000000001e-05 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666045+0000", 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014620861000000001 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666079+0000", 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3809000000000001e-05 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.543 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.826948272000003, 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015570855999999999, 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692736+0000", 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031561799999999999 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692809+0000", 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3174000000000001e-05 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692821+0000", 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1627000000000001e-05 2024-09-15T02:49:45.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692906+0000", 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5172000000000004e-05 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692920+0000", 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4361e-05 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707954+0000", 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015033451999999999 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707991+0000", 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6714000000000001e-05 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:45.548 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.769886116999999, 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014530774999999999, 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749483+0000", 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4799999999999997e-07 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749717+0000", 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000234408 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749768+0000", 2024-09-15T02:49:45.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1155000000000001e-05 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749777+0000", 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1609999999999992e-06 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749853+0000", 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5803000000000006e-05 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749872+0000", 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556e-05 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.763971+0000", 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014098934 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.764013+0000", 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1810000000000001e-05 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.552 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.265005690999999, 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012056918, 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254363+0000", 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7199999999999997e-07 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254896+0000", 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053267899999999999 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255034+0000", 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013821100000000001 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255066+0000", 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:45.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255135+0000", 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8885000000000002e-05 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255148+0000", 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2498e-05 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266383+0000", 2024-09-15T02:49:45.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011235187000000001 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266419+0000", 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6358e-05 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.228231086000001, 2024-09-15T02:49:45.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011905962000000001, 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291138+0000", 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291342+0000", 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203403 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291572+0000", 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230084 2024-09-15T02:49:45.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291579+0000", 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6990000000000003e-06 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291641+0000", 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2093000000000003e-05 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291650+0000", 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303018+0000", 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011368296 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303043+0000", 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4901000000000001e-05 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.227996695999998, 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011692335, 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291548+0000", 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000177021 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291673+0000", 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012469299999999999 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291678+0000", 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:45.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296939+0000", 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052609120000000004 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296948+0000", 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303051+0000", 2024-09-15T02:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006102433 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303064+0000", 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2948999999999999e-05 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/33 ECSubWrite(tid=212, reqid=client.4526.0:450, at_version=52'22, trim_to=0'0, roll_forward_to=49'21))", 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.270908513000002, 2024-09-15T02:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015878961, 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248460+0000", 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249338+0000", 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087768999999999998 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249433+0000", 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5333000000000004e-05 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249451+0000", 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8406e-05 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249656+0000", 2024-09-15T02:49:45.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020463100000000001 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249685+0000", 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8943000000000001e-05 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264308+0000", 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014623486999999999 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264338+0000", 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0020999999999999e-05 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.232710767, 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020897651, 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286658+0000", 2024-09-15T02:49:45.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.58e-07 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287075+0000", 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041690100000000001 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287318+0000", 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024316499999999999 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287324+0000", 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9580000000000004e-06 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287521+0000", 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197746 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287532+0000", 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366e-05 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307520+0000", 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019987953999999999 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307555+0000", 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5102999999999999e-05 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.575 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.232274015000002, 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020477789999999999, 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3200000000000001e-07 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287283+0000", 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018875200000000001 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287367+0000", 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4036000000000007e-05 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287374+0000", 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5340000000000002e-06 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295137+0000", 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077628599999999999 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295149+0000", 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307560+0000", 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012411447000000001 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307572+0000", 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-15T02:49:45.579 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=413, reqid=client.4526.0:673, at_version=72'80, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.226431187999999, 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015088048, 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8799999999999999e-07 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292992+0000", 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5479000000000002e-05 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293291+0000", 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029850500000000002 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293295+0000", 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4179999999999999e-06 2024-09-15T02:49:45.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302806+0000", 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095105680000000005 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302817+0000", 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1464999999999999e-05 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307993+0000", 2024-09-15T02:49:45.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051757540000000003 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308025+0000", 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1671000000000002e-05 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:c, version: 71'82, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:c@71'82, size: 1081344, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:45.584 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.804682497000002, 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013023088, 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714687+0000", 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.866e-06 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715107+0000", 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419293 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715195+0000", 2024-09-15T02:49:45.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8758999999999995e-05 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715215+0000", 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0143e-05 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715370+0000", 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000154611 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.727709+0000", 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012338416 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:head, version: 71'83, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:head@71'83, size: 2293760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.533845607, 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011824942, 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.588 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985524+0000", 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.533e-06 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985735+0000", 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021152599999999999 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985816+0000", 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0516e-05 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985833+0000", 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7413000000000001e-05 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985974+0000", 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141161 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.997347+0000", 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011372793000000001 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.591 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7f1f21a6:::smithi19125844-10:head, version: 71'85, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:7f1f21a6:::smithi19125844-10:head@71'85, size: 1736704, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.274749059000001, 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014391049, 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244620+0000", 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4300000000000001e-06 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245318+0000", 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069816099999999999 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245404+0000", 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5782e-05 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245423+0000", 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8272999999999999e-05 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245544+0000", 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121785 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.259010+0000", 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013465618 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=472, reqid=client.4526.0:793, at_version=79'84, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.035900869999999, 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013978862999999999, 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483468+0000", 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483595+0000", 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012675800000000001 2024-09-15T02:49:45.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483644+0000", 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9200000000000003e-05 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483653+0000", 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4030000000000007e-06 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483747+0000", 2024-09-15T02:49:45.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4239000000000004e-05 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483762+0000", 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5698000000000001e-05 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497313+0000", 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013550859 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497446+0000", 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013259400000000001 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=473, reqid=client.4526.0:794, at_version=79'85, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:45.600 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.029685641, 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014861180999999999, 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489684+0000", 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.282e-06 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490105+0000", 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042170800000000002 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490163+0000", 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8158000000000001e-05 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490173+0000", 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4350000000000003e-06 2024-09-15T02:49:45.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490301+0000", 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127928 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490314+0000", 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3104e-05 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504495+0000", 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014181545 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504543+0000", 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8021e-05 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 95/71 ECSubWrite(tid=539, reqid=client.4526.0:958, at_version=95'82, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.2487099339999999, 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025430995000000001, 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270659+0000", 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271073+0000", 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000414342 2024-09-15T02:49:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271106+0000", 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3107e-05 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271113+0000", 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5910000000000001e-06 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271176+0000", 2024-09-15T02:49:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3633999999999994e-05 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271187+0000", 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0436e-05 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296049+0000", 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024862042000000001 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296089+0000", 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0102000000000003e-05 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 95/71 ECSubWrite(tid=542, reqid=client.4526.0:959, at_version=95'83, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.2402664200000002, 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017183397, 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279111+0000", 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.251e-06 2024-09-15T02:49:45.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279556+0000", 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044511799999999999 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279593+0000", 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.663e-05 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279600+0000", 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1400000000000002e-06 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279671+0000", 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.161e-05 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279681+0000", 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7559999999999994e-06 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296269+0000", 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016588354 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296285+0000", 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5537999999999999e-05 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.614 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.616 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:45.616 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:45.616 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:45.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:45.617 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:45.612+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:45.717 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:45.716+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:45.724 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.724 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:45.724 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:45.724 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:45.818 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:45.816+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:45.833 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:45.835 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.835 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:45.835 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:45.835 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:45.835 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:45.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:45.859 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:45.890 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:45.918 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:45.919 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:45.916+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:45.947 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:45.950 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.583460990000006, 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:45.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.223889475999997, 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 80.378333032, 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:45.966 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:45.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 80.360814456, 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:45.974 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:45.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.546952028999996, 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:45.978 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:45.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:45.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 52.240708390999998, 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:45.985 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:45.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:45.990 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 52.240650557999999, 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:45.991 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:45.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:45.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:45.996 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 52.240615372000001, 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:45.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:46.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:46.002 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 47.505310813000001, 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:46.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:46.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 46.666878599999997, 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:46.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.010 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.011 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:670 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:46.011 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.663584738000001, 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027236222000000001, 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 670 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8800000000000003e-07 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283625+0000", 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3329999999999999e-06 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283630+0000", 2024-09-15T02:49:46.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7450000000000001e-06 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283638+0000", 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7349999999999996e-06 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283962+0000", 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032339899999999998 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284074+0000", 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011208499999999999 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298432+0000", 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014357939 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298517+0000", 2024-09-15T02:49:46.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:49:46.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310858+0000", 2024-09-15T02:49:46.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012341388999999999 2024-09-15T02:49:46.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.018 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.019 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.bs2 72/33 ECSubRead(tid=420, to_read={3:d2661d0e:::smithi19125844-43:head=0,4096,0}, subchunks={3:d2661d0e:::smithi19125844-43:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:46.019 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:46.019 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.636403398999999, 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085265647999999999, 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310804+0000", 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310814+0000", 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0115000000000001e-05 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395542+0000", 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.084728263999999998 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395552+0000", 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5899999999999997e-06 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395608+0000", 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5832999999999999e-05 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.396069+0000", 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046113399999999998 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.023 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:753 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.486454926, 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027324345, 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 753 2024-09-15T02:49:46.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460754+0000", 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9709999999999998e-06 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460760+0000", 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7510000000000002e-06 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460773+0000", 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2136999999999999e-05 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460780+0000", 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471867+0000", 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011087001000000001 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471873+0000", 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000002e-06 2024-09-15T02:49:46.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.484980+0000", 2024-09-15T02:49:46.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0131073370000001 2024-09-15T02:49:46.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485369+0000", 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038873500000000002 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488076+0000", 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027074099999999999 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:754 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.486419338000001, 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0276021209999999, 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 754 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460788+0000", 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8899999999999999e-07 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460791+0000", 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.411e-06 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460798+0000", 2024-09-15T02:49:46.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4679999999999997e-06 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460802+0000", 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2889999999999999e-06 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471896+0000", 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011093591 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471897+0000", 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.331e-06 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485874+0000", 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0139766539999999 2024-09-15T02:49:46.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485962+0000", 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8481999999999998e-05 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488389+0000", 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024273060000000002 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.037 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:755 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.486400321000001, 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0311918470000001, 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.038 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 755 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460807+0000", 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460809+0000", 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460815+0000", 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1249999999999998e-06 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460819+0000", 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9670000000000003e-06 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471918+0000", 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098856000000001 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471919+0000", 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488503+0000", 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0165840150000001 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488618+0000", 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011474199999999999 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491998+0000", 2024-09-15T02:49:46.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033803650000000002 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:766 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:46.044 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.486383291999999, 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0356599440000001, 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.045 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 766 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460824+0000", 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.27e-07 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460831+0000", 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460837+0000", 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-09-15T02:49:46.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460841+0000", 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5889999999999999e-06 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471939+0000", 2024-09-15T02:49:46.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098591 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471940+0000", 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488932+0000", 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0169917230000001 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.489075+0000", 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014318299999999999 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.496483+0000", 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074080589999999998 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:767 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.486362234000001, 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.037162342, 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 767 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:46.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460845+0000", 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3200000000000001e-07 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460847+0000", 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.554e-06 2024-09-15T02:49:46.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460853+0000", 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020000000000002e-06 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460857+0000", 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0679999999999998e-06 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471961+0000", 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011103755999999999 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471962+0000", 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3600000000000003e-07 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.490940+0000", 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.018977662 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491003+0000", 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3081000000000001e-05 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.498007+0000", 2024-09-15T02:49:46.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070041510000000001 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:768 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.486345316000001, 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.038324754, 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.057 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 768 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460862+0000", 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.34e-07 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460864+0000", 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6639999999999998e-06 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460870+0000", 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0859999999999998e-06 2024-09-15T02:49:46.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460874+0000", 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.563e-06 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471982+0000", 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011108122999999999 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471983+0000", 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6700000000000003e-07 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492584+0000", 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0206010889999999 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492705+0000", 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012115100000000001 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.499186+0000", 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006481077 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 95/77 ECSubWriteReply(tid=293, last_complete=95'119, committed=1, applied=1))", 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.6838151809999999, 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55831691699999997, 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263392+0000", 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263396+0000", 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0960000000000003e-06 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821430+0000", 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55803327199999997 2024-09-15T02:49:46.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821437+0000", 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8320000000000006e-06 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821583+0000", 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014531099999999999 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821708+0000", 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012571500000000001 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.es1 96/76 ECSubWrite(tid=159, reqid=osd.2.0:159, at_version=96'126, trim_to=0'0, roll_forward_to=96'124))", 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:40.308655+0000", 2024-09-15T02:49:46.068 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 5.638551788, 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062686333999999996, 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308655+0000", 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308655+0000", 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308656+0000", 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.75e-07 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308678+0000", 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2728e-05 2024-09-15T02:49:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369529+0000", 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.060850298999999997 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369546+0000", 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7263000000000001e-05 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369648+0000", 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102416 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369665+0000", 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7062e-05 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.371306+0000", 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016409829999999999 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.371341+0000", 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4907999999999998e-05 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.073 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:46.075 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:46.075 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:46.075 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:46.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:46.076 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:46.067+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:46.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:46.087 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:46.083+0000 7fa93effd700 0 -- 172.21.15.6:0/663608985 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7fa940193130 msgr2=0x7fa940195510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:46.176 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:46.171+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:46.185 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:46.186 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:46.186 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:46.186 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:46.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:46.277 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:46.276+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:46.297 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:46.298 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:46.298 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:46.298 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:46.298 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:46.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:46.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T02:49:46.378 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:46.376+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:46.409 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:46.409 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:46.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:46.409 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/33 ECSubWriteReply(tid=19, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 63.867444974999998, 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74030938599999996, 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539611+0000", 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3680000000000001e-06 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539619+0000", 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-09-15T02:49:46.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278674+0000", 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73905563200000002 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278687+0000", 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.239e-05 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278784+0000", 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7149000000000001e-05 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.279919+0000", 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001135645 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=24, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 62.820077685999998, 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30338358999999998, 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586978+0000", 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.51e-07 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586981+0000", 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890125+0000", 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30314395700000002 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890138+0000", 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2627e-05 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890297+0000", 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015958899999999999 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890361+0000", 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3658999999999995e-05 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.417 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:414 3.3s0 3:cebed992:::smithi19125844-27:head [copy-from ver 4] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 52.151191251999997, 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034868557000000001, 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 414 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255866+0000", 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.621e-06 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255879+0000", 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2738999999999999e-05 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255912+0000", 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3099000000000003e-05 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255927+0000", 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5013e-05 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256006+0000", 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8688000000000003e-05 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256016+0000", 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8379999999999997e-06 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262150+0000", 2024-09-15T02:49:46.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061347040000000004 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262292+0000", 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141595 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.275991+0000", 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013699266999999999 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282717+0000", 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067257910000000001 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290669+0000", 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079519480000000003 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290732+0000", 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3254000000000001e-05 2024-09-15T02:49:46.423 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=77, last_complete=68'67, committed=1, applied=1))", 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.994694211000002, 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.106159008, 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.424 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412362+0000", 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9999999999999996e-07 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412370+0000", 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4120000000000001e-06 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517947+0000", 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10557704800000001 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517957+0000", 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0173999999999999e-05 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518475+0000", 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051777099999999999 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518520+0000", 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4703000000000002e-05 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.427 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:636 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~606208] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.910039064000003, 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026984608, 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.428 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 636 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497019+0000", 2024-09-15T02:49:46.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.066e-06 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497546+0000", 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052673800000000001 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497580+0000", 2024-09-15T02:49:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4647999999999998e-05 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497592+0000", 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1669e-05 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497695+0000", 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103031 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497915+0000", 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219607 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.499984+0000", 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020692800000000002 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518517+0000", 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018533353999999998 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523868+0000", 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053511779999999998 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524001+0000", 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132037 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:638 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~475136] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.898939464999998, 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.170696291, 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 638 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:46.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508118+0000", 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6110000000000001e-06 2024-09-15T02:49:46.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508908+0000", 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00078945100000000004 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508947+0000", 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9298999999999998e-05 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508971+0000", 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4329e-05 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518663+0000", 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0096915070000000002 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518760+0000", 2024-09-15T02:49:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7740999999999998e-05 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.519373+0000", 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061209299999999997 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523608+0000", 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042356319999999996 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678746+0000", 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15513796099999999 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678812+0000", 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5667000000000007e-05 2024-09-15T02:49:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:639 3.3s0 3:c4165a1c:::smithi19125844-22:head [setxattr _header (57) in=7b] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.898054942000002, 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.189219685, 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.442 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 639 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509002+0000", 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7579999999999999e-06 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509010+0000", 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3110000000000006e-06 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509031+0000", 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1234e-05 2024-09-15T02:49:46.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509052+0000", 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0962e-05 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522387+0000", 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013334693 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522514+0000", 2024-09-15T02:49:46.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126723 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523057+0000", 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054339999999999998 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524433+0000", 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001375739 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698072+0000", 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17363879099999999 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698220+0000", 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014807399999999999 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:640 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:46.447 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.897967711, 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19189399200000001, 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 640 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:46.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509090+0000", 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2800000000000002e-06 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509095+0000", 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9679999999999997e-06 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509113+0000", 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8031000000000001e-05 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509122+0000", 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6509999999999996e-06 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524120+0000", 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014997610999999999 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524399+0000", 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027887000000000002 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698989+0000", 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.174589943 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.699138+0000", 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149333 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.700981+0000", 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018433049999999999 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=80, last_complete=68'71, committed=1, applied=1))", 2024-09-15T02:49:46.452 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 33.879841183000003, 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15161024100000001, 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:46.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527215+0000", 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.088e-06 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527221+0000", 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9909999999999999e-06 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678531+0000", 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15131061000000001 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678545+0000", 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3597e-05 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678666+0000", 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012109599999999999 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678824+0000", 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015785899999999999 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:699 3.12s0 3:4a71560d:::smithi19125844-16:head [rollback 24] snapc 26=[26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.938316371999999, 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.064459370000000002, 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.456 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 699 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468741+0000", 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0420000000000001e-06 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468749+0000", 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000002e-06 2024-09-15T02:49:46.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468764+0000", 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5353999999999999e-05 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468773+0000", 2024-09-15T02:49:46.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7649999999999999e-06 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495474+0000", 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026701740000000002 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495855+0000", 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038018500000000001 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512214+0000", 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016359598 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512474+0000", 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025961499999999999 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.533198+0000", 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020724131 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.462 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:703 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.93827456, 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056538106999999997, 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.463 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 703 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468781+0000", 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468786+0000", 2024-09-15T02:49:46.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4909999999999999e-06 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468794+0000", 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0229999999999992e-06 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468803+0000", 2024-09-15T02:49:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1910000000000006e-06 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.500871+0000", 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032068169000000001 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502024+0000", 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001152674 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525089+0000", 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023065130999999999 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525319+0000", 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022968700000000001 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.468 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:704 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.938240624999999, 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056656818999999997, 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.469 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 704 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468815+0000", 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4199999999999995e-07 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468819+0000", 2024-09-15T02:49:46.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7359999999999999e-06 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468826+0000", 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0190000000000001e-06 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468829+0000", 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5499999999999999e-06 2024-09-15T02:49:46.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502105+0000", 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033275546000000003 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502234+0000", 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012945 2024-09-15T02:49:46.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525355+0000", 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023120971000000001 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525471+0000", 2024-09-15T02:49:46.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011580499999999999 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:705 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.938220948000001, 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056845816, 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.475 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 705 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:46.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468838+0000", 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0690000000000002e-06 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468843+0000", 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7219999999999996e-06 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468849+0000", 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0340000000000002e-06 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502534+0000", 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033684990999999997 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502672+0000", 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137701 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525531+0000", 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022859147 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525680+0000", 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014871700000000001 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.479 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 83/71 ECSubWriteReply(tid=154, last_complete=83'94, committed=1, applied=1))", 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.817519643000001, 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54901637599999997, 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589536+0000", 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6800000000000002e-07 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589538+0000", 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.497e-06 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138412+0000", 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54887397000000004 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138426+0000", 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3621999999999999e-05 2024-09-15T02:49:46.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138495+0000", 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9417000000000003e-05 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138552+0000", 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6601999999999998e-05 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:901 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~638976] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.277952+0000", 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.129103082, 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.036249844000000003, 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 901 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.277952+0000", 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.277952+0000", 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.277955+0000", 2024-09-15T02:49:46.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4079999999999998e-06 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278491+0000", 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053570099999999997 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278520+0000", 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8688e-05 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278530+0000", 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8519999999999999e-06 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278683+0000", 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015307400000000001 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278877+0000", 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019389599999999999 2024-09-15T02:49:46.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280761+0000", 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001884658 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.300773+0000", 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020011856000000001 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314122+0000", 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013348956 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314202+0000", 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9754999999999998e-05 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:902 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~671744] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:46.489 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.281962+0000", 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.125093006, 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034372422, 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 902 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281962+0000", 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281962+0000", 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281964+0000", 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1090000000000001e-06 2024-09-15T02:49:46.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.282424+0000", 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046013199999999998 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.282447+0000", 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2671999999999999e-05 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.282455+0000", 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9540000000000004e-06 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.291848+0000", 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093927769999999997 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.291993+0000", 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014574600000000001 2024-09-15T02:49:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.293159+0000", 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001165346 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.312623+0000", 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019464250999999998 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316207+0000", 2024-09-15T02:49:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035840730000000001 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316334+0000", 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127362 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:903 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~802816] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287154+0000", 2024-09-15T02:49:46.495 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.119901350999999, 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.029421439000000001, 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 903 2024-09-15T02:49:46.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287154+0000", 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287154+0000", 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287155+0000", 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.773e-06 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287738+0000", 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00058255399999999999 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287759+0000", 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0987000000000001e-05 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287770+0000", 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1035e-05 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.301023+0000", 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013252788 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.301121+0000", 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.87e-05 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302043+0000", 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00092160199999999999 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314264+0000", 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012220846 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316454+0000", 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021897890000000001 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316575+0000", 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121365 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.501 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:904 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~81920] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287779+0000", 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.119275678999999, 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.031042132, 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.502 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 904 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287779+0000", 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287779+0000", 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287780+0000", 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287817+0000", 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6891999999999998e-05 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287831+0000", 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3550000000000001e-05 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287835+0000", 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-06 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.312677+0000", 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024841196999999999 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.312803+0000", 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012656899999999999 2024-09-15T02:49:46.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313234+0000", 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000431261 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.315040+0000", 2024-09-15T02:49:46.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018058589999999999 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.318776+0000", 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037358999999999999 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.318821+0000", 2024-09-15T02:49:46.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5259999999999997e-05 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:905 3.15s0 3:a906e5df:::smithi19125844-36:head [setxattr _header (57) in=7b] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.119213117999999, 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.031686819999999997, 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:46.509 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 905 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:46.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3000000000000001e-07 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287846+0000", 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3859999999999999e-06 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287851+0000", 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5439999999999998e-06 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287855+0000", 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.264e-06 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314422+0000", 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026567048999999999 2024-09-15T02:49:46.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314491+0000", 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9197999999999997e-05 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314867+0000", 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037581300000000002 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316898+0000", 2024-09-15T02:49:46.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020310419999999998 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319422+0000", 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002524199 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319529+0000", 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106795 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:906 3.15s0 3:a906e5df:::smithi19125844-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287858+0000", 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.119196831, 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033687184000000002, 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 906 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287858+0000", 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287858+0000", 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287859+0000", 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7099999999999997e-07 2024-09-15T02:49:46.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287860+0000", 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.968e-06 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287866+0000", 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9000000000000003e-06 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287870+0000", 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4479999999999999e-06 2024-09-15T02:49:46.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.315082+0000", 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027211803999999999 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:46.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.315179+0000", 2024-09-15T02:49:46.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7507e-05 2024-09-15T02:49:46.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.320028+0000", 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048489040000000002 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.320161+0000", 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132829 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.321545+0000", 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013844529999999999 2024-09-15T02:49:46.520 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.521 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:46.521 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:46.521 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:46.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:46.523 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:46.515+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:46.533 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14, 15, 10] out_osds: [6] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 14, 3, 7, 11, 15, 10, 6] 2024-09-15T02:49:46.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:49:46.534 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-15T02:49:46.534 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T02:49:46.534 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T02:49:46.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-15T02:49:46.623 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:46.622+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:46.632 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:46.632 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:46.641 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:46.643 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:46.644 INFO:teuthology.orchestra.run.smithi118.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T02:49:46.644 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:46.725 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:46.724+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:46.741 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:46.741 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:46.751 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:46.826 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:46.824+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:46.847 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:46.848 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:46.858 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:46.927 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:46.924+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:46.960 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:46.960 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:46.970 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:49:47.028 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:47.024+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:47.072 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.072 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.081 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:49:47.129 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:47.128+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:47.182 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.182 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:47.230 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:47.226+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:47.236 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:47.268 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:47.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:47.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:47.302 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.302 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:47.330 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:47.326+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:47.337 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-15T02:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:47.431 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:47.423+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:47.525 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.525 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.532 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:47.532+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:47.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:47.632 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:47.627+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:47.679 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.679 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:47.734 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:47.730+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:47.791 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.792 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:47.834 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:47.832+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:47.903 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:47.903 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:47.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:47.936 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:47.932+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:48.014 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.014 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:48.036 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:48.034+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:48.129 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.129 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.137 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:48.134+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:48.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:48.238 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:48.231+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:48.238 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.238 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:48.338 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:48.336+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:48.348 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.348 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:48.439 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:48.435+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:48.455 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.455 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:48.541 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:48.540+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:48.617 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.617 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:48.642 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:48.638+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:48.738 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.738 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.743 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:48.740+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:48.747 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:48.844 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:48.840+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:48.845 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.845 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.854 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:48.944 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:48.939+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:48.955 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:48.955 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:48.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:49.045 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:49.044+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:49.067 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.067 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:49.146 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:49.143+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:49.176 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.176 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.185 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:49.247 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:49.244+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:49.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T02:49:49.299 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.299 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.308 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:49.348 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-15T02:49:49.343+0000 7fe72df36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 22126) UID: 0 2024-09-15T02:49:49.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T02:49:49.401 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.401 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.409 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:49.443 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:49.449 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:49.443+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:49.469 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:49.499 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:49.510 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.510 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:49.550 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:49.543+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:49.576 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:49.616 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.616 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.617 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:49.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:49.651 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:49.643+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:49.651 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:49.680 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:49.708 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:49.721 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.721 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:49.733 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:49.766 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:49.776 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:49.776+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:49.794 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:49.820 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:49.831 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:49.832 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:49.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:49.847 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:49.843+0000 7fd5377fe700 0 -- 172.21.15.6:0/1456328149 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7fd5381955c0 msgr2=0x7fd5381979a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T02:49:49.853 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:49.852+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:49.945 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-15T02:49:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-15T02:49:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-15T02:49:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-15T02:49:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:388 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.188971785, 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.205927366, 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 388 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752800+0000", 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752801+0000", 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752805+0000", 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8480000000000003e-06 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752820+0000", 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752827+0000", 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8059999999999997e-06 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770449+0000", 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017621527000000001 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770567+0000", 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118576 2024-09-15T02:49:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776872+0000", 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063044340000000003 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.776986+0000", 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011429699999999999 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958728+0000", 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.181741654 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:389 3.7s0 3:e10d9d4d:::smithi19125844-8:head [copy-from ver 4] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.188936523999999, 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47945425899999999, 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 389 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752835+0000", 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752836+0000", 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4600000000000004e-07 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752842+0000", 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8579999999999996e-06 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752860+0000", 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7816e-05 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752867+0000", 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0260000000000002e-06 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770793+0000", 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017925963999999999 2024-09-15T02:49:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.770931+0000", 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137978 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.955584+0000", 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.184652765 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.959571+0000", 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003987488 2024-09-15T02:49:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232070+0000", 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.272499243 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.232290+0000", 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021937500000000001 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:390 3.7s0 3:e10d9d4d:::smithi19125844-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.188898334000001, 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20558775300000001, 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 390 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752874+0000", 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8199999999999999e-07 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752878+0000", 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4489999999999999e-06 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752888+0000", 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0122000000000001e-05 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.752894+0000", 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9220000000000002e-06 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771143+0000", 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018248665000000001 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.771198+0000", 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5158000000000003e-05 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958258+0000", 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18706018699999999 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958359+0000", 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101025 2024-09-15T02:49:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.958461+0000", 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102543 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:402 3.1s0 3:801a1da7:::smithi19125844-39:e [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.183520571000003, 2024-09-15T02:49:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52960550699999998, 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 402 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758251+0000", 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758252+0000", 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5000000000000002e-07 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758257+0000", 2024-09-15T02:49:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3800000000000004e-06 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758267+0000", 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0455000000000001e-05 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.758604+0000", 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033705300000000003 2024-09-15T02:49:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761065+0000", 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024613840000000001 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.761208+0000", 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014282499999999999 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287857+0000", 2024-09-15T02:49:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52664865999999999 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 49/33 ECSubReadReply(tid=183, attrs_read=0))", 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.178548071000002, 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52471420199999996, 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763224+0000", 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999999e-07 2024-09-15T02:49:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:53.763572+0000", 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034791800000000001 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286378+0000", 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.52280558799999999 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286384+0000", 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.286558+0000", 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017370499999999999 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.287938+0000", 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013805550000000001 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:433 3.7s0 3:ec23d144:::smithi19125844-41:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.658778697999999, 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27191602199999998, 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282993+0000", 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.282994+0000", 2024-09-15T02:49:49.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1790000000000001e-06 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283000+0000", 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0479999999999998e-06 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283008+0000", 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1440000000000003e-06 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283012+0000", 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7129999999999996e-06 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298201+0000", 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015188495 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298355+0000", 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015441200000000001 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.298951+0000", 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059618300000000002 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300677+0000", 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001725285 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554804+0000", 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25412710900000002 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554909+0000", 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105454 2024-09-15T02:49:49.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:434 3.7s0 3:ec23d144:::smithi19125844-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.658754774999998, 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27651220700000001, 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 434 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283017+0000", 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283018+0000", 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0500000000000003e-07 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283021+0000", 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.959e-06 2024-09-15T02:49:49.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283028+0000", 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0169999999999999e-06 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283032+0000", 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0509999999999998e-06 2024-09-15T02:49:49.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299373+0000", 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016340659 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299481+0000", 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108218 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554991+0000", 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25551037700000001 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.555256+0000", 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026495400000000002 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559529+0000", 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042733670000000001 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.983 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:437 3.6s0 3:6e9f7d10:::smithi19125844-33:head [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.658687837999999, 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20344801100000001, 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.984 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 437 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283084+0000", 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283085+0000", 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283088+0000", 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.993e-06 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283094+0000", 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5810000000000002e-06 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283098+0000", 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6289999999999999e-06 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284035+0000", 2024-09-15T02:49:49.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093763400000000002 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.284100+0000", 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4882999999999995e-05 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486532+0000", 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.20243185599999999 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.988 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:438 3.7s0 3:ee3b3dd3:::smithi19125844-46:head [delete] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e50)", 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.658654028999997, 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27354936000000002, 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 438 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.989 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283118+0000", 2024-09-15T02:49:49.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8100000000000003e-07 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283121+0000", 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8830000000000002e-06 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283129+0000", 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3390000000000004e-06 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.283133+0000", 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299523+0000", 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016389989000000001 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.299625+0000", 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010208 2024-09-15T02:49:49.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300116+0000", 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049149000000000003 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.300972+0000", 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085554200000000004 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556582+0000", 2024-09-15T02:49:49.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25561025399999998 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.556667+0000", 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5208000000000001e-05 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.995 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.6s0 50/33 ECSubReadReply(tid=200, attrs_read=0))", 2024-09-15T02:49:49.996 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.646484436000001, 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19127902199999999, 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295288+0000", 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2600000000000001e-07 2024-09-15T02:49:49.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.295338+0000", 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9951999999999999e-05 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485520+0000", 2024-09-15T02:49:49.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.19018270600000001 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485530+0000", 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0820000000000005e-06 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.485655+0000", 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012580999999999999 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.486567+0000", 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000911246 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=203, last_complete=50'62, committed=1, applied=1))", 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.638439253999998, 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25159346799999999, 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.001 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303333+0000", 2024-09-15T02:49:50.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.303336+0000", 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9550000000000001e-06 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554637+0000", 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.25130088299999997 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554647+0000", 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9329999999999996e-06 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554704+0000", 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6514000000000001e-05 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.554926+0000", 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222647 2024-09-15T02:49:50.004 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 50/33 ECSubWriteReply(tid=205, last_complete=50'35, committed=1, applied=1))", 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.622431824000003, 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.23186345, 2024-09-15T02:49:50.005 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.008 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.008 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:50.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319340+0000", 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319341+0000", 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6700000000000003e-07 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.319347+0000", 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6879999999999996e-06 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550925+0000", 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.231578591 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.550946+0000", 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0903e-05 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551110+0000", 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016408300000000001 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.551204+0000", 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3418000000000006e-05 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 50/33 ECSubWriteReply(tid=208, last_complete=50'64, committed=1, applied=1))", 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.382574658000003, 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.60425413100000003, 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559197+0000", 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559198+0000", 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9500000000000001e-07 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:54.559203+0000", 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.13e-06 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163222+0000", 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.604018638 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163237+0000", 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4622e-05 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163384+0000", 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014714099999999999 2024-09-15T02:49:50.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:55.163451+0000", 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7805000000000006e-05 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 54/53 ECSubWrite(tid=87, reqid=client.4526.0:461, at_version=0'0, trim_to=0'0, roll_forward_to=54'39))", 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.020671991, 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.41018363000000002, 2024-09-15T02:49:50.015 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921100+0000", 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921102+0000", 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:58.921127+0000", 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4987e-05 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329820+0000", 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.40869367400000001 2024-09-15T02:49:50.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329833+0000", 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2669e-05 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329958+0000", 2024-09-15T02:49:50.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012490099999999999 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329981+0000", 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2463000000000002e-05 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331170+0000", 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001189701 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.331284+0000", 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113435 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDPGPushReply(3.1s0 54/53 [PushReplyOp(3:801a1da7:::smithi19125844-39:head)])", 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.646955208000001, 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99244340499999995, 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.020 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294817+0000", 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294818+0000", 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.294837+0000", 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9052999999999999e-05 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286615+0000", 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99177708899999995 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286629+0000", 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4583000000000001e-05 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.286721+0000", 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1640999999999999e-05 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:00.287260+0000", 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053939199999999997 2024-09-15T02:49:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpRead(3.9s2 55/53 ECSubRead(tid=88, to_read={3:94f52c52:::smithi19125844-20:head=0,110592,0}, subchunks={3:94f52c52:::smithi19125844-20:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.611884734999997, 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59246917700000001, 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329887+0000", 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329888+0000", 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-09-15T02:49:50.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.329900+0000", 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.188e-05 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921816+0000", 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59191607099999999 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.921831+0000", 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4765e-05 2024-09-15T02:49:50.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922011+0000", 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018023900000000001 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:48:59.922356+0000", 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034564200000000001 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 62/53 ECSubWriteReply(tid=316, last_complete=62'23, committed=1, applied=1))", 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.584330238, 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58891104599999999, 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:50.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357442+0000", 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6400000000000002e-07 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.357447+0000", 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9529999999999998e-06 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945262+0000", 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58781430300000004 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945281+0000", 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9366000000000001e-05 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.945429+0000", 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014825600000000001 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:06.946353+0000", 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092350400000000001 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.0.22:2 3.7s0 3:e10d9d4d:::smithi19125844-8:head [assert-version v25,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:50.031 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.465854547999999, 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.014934445, 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475917+0000", 2024-09-15T02:49:50.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475920+0000", 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475928+0000", 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1750000000000005e-06 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475952+0000", 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4244e-05 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.475960+0000", 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0940000000000008e-06 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476051+0000", 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1032000000000005e-05 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:21.476057+0000", 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3029999999999999e-06 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484706+0000", 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0086496460000001 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.484954+0000", 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024785999999999997 2024-09-15T02:49:50.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:22.490852+0000", 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058976979999999998 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4526.0:912 3.bs0 3:d6271403:::smithi19125844-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.626275080999999, 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.18619464499999999, 2024-09-15T02:49:50.040 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4526", 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 912 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315497+0000", 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7899999999999998e-07 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315501+0000", 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2399999999999999e-06 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315519+0000", 2024-09-15T02:49:50.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8094999999999999e-05 2024-09-15T02:49:50.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.315525+0000", 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8429999999999998e-06 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.340958+0000", 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025432834000000001 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.341123+0000", 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016572000000000001 2024-09-15T02:49:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.499134+0000", 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.15801039 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.499395+0000", 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026174100000000001 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.501692+0000", 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022962030000000001 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 88/76 ECSubWriteReply(tid=531, last_complete=88'112, committed=1, applied=1))", 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:50.046 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.441026303000001, 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74370967399999999, 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:50.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500746+0000", 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7700000000000004e-07 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:32.500750+0000", 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8229999999999997e-06 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244204+0000", 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74345404000000004 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244216+0000", 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1423000000000001e-05 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244348+0000", 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000132464 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-15T02:49:33.244455+0000", 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107347 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-15T02:49:50.050 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-15T02:49:50.053 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:50.054 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:50.052+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:50.153 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:50.147+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:50.164 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:50.164 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:50.164 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:50.164 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:50.173 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:50.255 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:50.252+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:50.281 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:50.281 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:50.281 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:50.281 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:50.281 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:50.289 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:50.350 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-15T02:49:50.355 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:50.352+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:50.392 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_repop(client.4189.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'21)", 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 85.652447589000005, 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.412456139, 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.393 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736281+0000", 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736282+0000", 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3039999999999999e-06 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.394 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:24.736295+0000", 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2819999999999999e-05 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147329+0000", 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.41103379899999998 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147340+0000", 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0859999999999999e-05 2024-09-15T02:49:50.395 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147465+0000", 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000125568 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.147535+0000", 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9431999999999995e-05 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.396 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148658+0000", 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011236869999999999 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:25.148737+0000", 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8669000000000002e-05 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:403 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~606208] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:50.397 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.631708895000003, 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.195887642, 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 403 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:50.398 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757019+0000", 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757022+0000", 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.593e-06 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.399 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757800+0000", 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077824999999999997 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757830+0000", 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0057e-05 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757844+0000", 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3947999999999999e-05 2024-09-15T02:49:50.400 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.757925+0000", 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0249999999999999e-05 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.758082+0000", 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015777599999999999 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.759386+0000", 2024-09-15T02:49:50.401 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001303217 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777348+0000", 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017962568000000002 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952681+0000", 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17533264000000001 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.402 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952907+0000", 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000226343 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:404 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~720896] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.627490291000001, 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.192160255, 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 404 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761238+0000", 2024-09-15T02:49:50.404 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.761280+0000", 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.2138999999999998e-05 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762400+0000", 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001119451 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.405 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762433+0000", 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.3238000000000003e-05 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762446+0000", 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3308e-05 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768118+0000", 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056721360000000004 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.768285+0000", 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000166482 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.769662+0000", 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013768140000000001 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780475+0000", 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010813578000000001 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.407 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953275+0000", 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17280025199999999 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953398+0000", 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012285700000000001 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.408 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:405 3.es0 3:71c1a91c:::smithi19125844-45:head [append 0~49152] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.626259552999997, 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19129260000000001, 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 405 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.409 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762469+0000", 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762470+0000", 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4699999999999999e-06 2024-09-15T02:49:50.410 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762527+0000", 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7151999999999998e-05 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762550+0000", 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2311999999999999e-05 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762559+0000", 2024-09-15T02:49:50.411 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9400000000000008e-06 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777614+0000", 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015055628999999999 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.777696+0000", 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1840000000000002e-05 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.412 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778120+0000", 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042372900000000002 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780527+0000", 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00240712 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953622+0000", 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17309514500000001 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.413 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.953761+0000", 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013926300000000001 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:406 3.es0 3:71c1a91c:::smithi19125844-45:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.626159592999997, 2024-09-15T02:49:50.414 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19225968700000001, 2024-09-15T02:49:50.415 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 406 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.416 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762569+0000", 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762570+0000", 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.034e-06 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762575+0000", 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4079999999999997e-06 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.417 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762592+0000", 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.715e-05 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762600+0000", 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6480000000000003e-06 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779660+0000", 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017060035000000001 2024-09-15T02:49:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.779757+0000", 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6537000000000001e-05 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.780137+0000", 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038070299999999998 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.419 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781414+0000", 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0012764230000000001 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954649+0000", 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17323481900000001 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.954828+0000", 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017992999999999999 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.420 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:407 3.es0 3:71c1a91c:::smithi19125844-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.626119721000002, 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.19532122900000001, 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 407 2024-09-15T02:49:50.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762609+0000", 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.422 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762610+0000", 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2200000000000002e-07 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762614+0000", 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.0659999999999997e-06 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762626+0000", 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2490000000000001e-05 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.423 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.762639+0000", 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2704000000000001e-05 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.781806+0000", 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019166823999999999 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.782082+0000", 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027635000000000002 2024-09-15T02:49:50.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955680+0000", 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17359843999999999 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.955807+0000", 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012658299999999999 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.957930+0000", 2024-09-15T02:49:50.425 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0021228499999999999 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.13.22:5 3.es0 3:740f3d5b:::smithi19125844-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.613588741000001, 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.177222558, 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.426 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775140+0000", 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775141+0000", 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.59e-06 2024-09-15T02:49:50.427 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775146+0000", 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.5630000000000002e-06 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775159+0000", 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3509999999999999e-05 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.428 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.775165+0000", 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8130000000000001e-06 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778738+0000", 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003572867 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.778948+0000", 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00021028299999999999 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.429 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:53.952362+0000", 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.17341393199999999 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:529 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~802816] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 46.133903283999999, 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.030230086, 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.430 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 529 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254825+0000", 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.431 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.254827+0000", 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6390000000000001e-06 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256167+0000", 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0013402900000000001 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.432 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256196+0000", 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8762999999999999e-05 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.256203+0000", 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6229999999999997e-06 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259870+0000", 2024-09-15T02:49:50.433 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0036663860000000002 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.260012+0000", 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014185099999999999 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.261784+0000", 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0017728349999999999 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.434 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277478+0000", 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015693650999999999 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285001+0000", 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075225120000000003 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285055+0000", 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4536000000000001e-05 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.435 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:530 3.es0 3:7f1f21a6:::smithi19125844-10:head [append 0~638976] snapc 18=[18,15,14,11,10,b,a] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 46.129982130000002, 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026488489, 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.436 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 530 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258746+0000", 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.258748+0000", 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.66e-06 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259495+0000", 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00074681400000000005 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259522+0000", 2024-09-15T02:49:50.438 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7552999999999999e-05 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259538+0000", 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6220000000000001e-05 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269841+0000", 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010302477000000001 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.439 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.269927+0000", 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5993000000000002e-05 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.270621+0000", 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00069455099999999996 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277769+0000", 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0071479769999999998 2024-09-15T02:49:50.440 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285179+0000", 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074099320000000002 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.285235+0000", 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5312000000000002e-05 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.441 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:533 3.es0 3:7f1f21a6:::smithi19125844-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 46.129023435999997, 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027459596999999999, 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 533 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.442 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259705+0000", 2024-09-15T02:49:50.443 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.89e-07 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259707+0000", 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5460000000000001e-06 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259711+0000", 2024-09-15T02:49:50.444 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1439999999999996e-06 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.259715+0000", 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3340000000000003e-06 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277889+0000", 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018173363000000001 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.445 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.277948+0000", 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8913999999999999e-05 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286109+0000", 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0081616199999999996 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.286156+0000", 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6928000000000002e-05 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.446 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.287165+0000", 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010083589999999999 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s2 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.854627306000001, 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027643175999999998, 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.447 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534101+0000", 2024-09-15T02:49:50.448 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534111+0000", 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.8090000000000004e-06 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.449 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534425+0000", 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031419599999999998 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542230+0000", 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0078045069999999996 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542239+0000", 2024-09-15T02:49:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542375+0000", 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000135897 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.542396+0000", 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1237999999999999e-05 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.451 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561688+0000", 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019291689000000001 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.561744+0000", 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6178999999999999e-05 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.452 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpRead(3.8s2 77/76 ECSubRead(tid=258, to_read={3:173bfaf3:::smithi19125844-44:18=262144,262144,0}, subchunks={3:173bfaf3:::smithi19125844-44:18=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.812868548000001, 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119859681, 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.453 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575860+0000", 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575861+0000", 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0640000000000001e-06 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.575876+0000", 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5165e-05 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.454 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694960+0000", 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.119084367 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.694980+0000", 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9930000000000001e-05 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695069+0000", 2024-09-15T02:49:50.455 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8737000000000001e-05 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.695719+0000", 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00065041800000000005 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=112, attrs_read=0))", 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:50.456 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.808378515000001, 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13363014300000001, 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580350+0000", 2024-09-15T02:49:50.457 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580351+0000", 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2419999999999999e-06 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.580721+0000", 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00037001299999999998 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.458 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712609+0000", 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.13188792199999999 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712623+0000", 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.429e-05 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.712782+0000", 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000158791 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.459 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.713980+0000", 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011978850000000001 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.es0 77/76 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.641351270000001, 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21641586800000001, 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.460 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747377+0000", 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.461 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747379+0000", 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.747806+0000", 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042687399999999999 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962958+0000", 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.21515266799999999 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.462 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.962974+0000", 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6237999999999999e-05 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963059+0000", 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4968000000000004e-05 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.963793+0000", 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00073351999999999998 2024-09-15T02:49:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.12s5 78/77 ECSubWrite(tid=129, reqid=client.4526.0:743, at_version=78'41, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 27.866886225000002, 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028389666000000001, 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521842+0000", 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.521843+0000", 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.122e-06 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.465 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522442+0000", 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00059853399999999996 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533671+0000", 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011229075 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533686+0000", 2024-09-15T02:49:50.466 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5265999999999999e-05 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533798+0000", 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000112198 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.533821+0000", 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.2229999999999999e-05 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.467 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550165+0000", 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016344826 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.550232+0000", 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6414999999999993e-05 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.468 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:836 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~573440] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.790091928999999, 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033215366000000003, 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 836 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.469 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598636+0000", 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.598639+0000", 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1280000000000002e-06 2024-09-15T02:49:50.470 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599731+0000", 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001092469 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599759+0000", 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8472000000000001e-05 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.599774+0000", 2024-09-15T02:49:50.471 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4612e-05 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608764+0000", 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089903399999999994 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.608966+0000", 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00020188699999999999 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.472 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.610666+0000", 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0016996940000000001 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.630970+0000", 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020303799000000001 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631771+0000", 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00080119100000000003 2024-09-15T02:49:50.473 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631852+0000", 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0773999999999996e-05 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:837 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~688128] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.785176450000002, 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037719309999999999, 2024-09-15T02:49:50.474 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 837 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.475 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603552+0000", 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.603553+0000", 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5999999999999999e-06 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604429+0000", 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000875577 2024-09-15T02:49:50.476 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604454+0000", 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4794999999999999e-05 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.604464+0000", 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9199999999999999e-06 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621603+0000", 2024-09-15T02:49:50.477 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017138823000000001 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.621666+0000", 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2996000000000005e-05 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.622382+0000", 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00071611400000000001 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.478 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631868+0000", 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0094864259999999992 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641117+0000", 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0092490960000000001 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641271+0000", 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015396299999999999 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.479 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:838 3.es0 3:7773fe95:::smithi19125844-37:head [append 0~294912] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.783234262000001, 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.03629342, 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.480 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 838 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605494+0000", 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.481 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605496+0000", 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.415e-06 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605778+0000", 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00028258100000000001 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605797+0000", 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8817999999999998e-05 2024-09-15T02:49:50.482 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605805+0000", 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5229999999999995e-06 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626744+0000", 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020938365 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.626872+0000", 2024-09-15T02:49:50.483 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012857999999999999 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.627645+0000", 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00077218100000000004 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631967+0000", 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0043226499999999999 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.484 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641643+0000", 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096753510000000004 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.641788+0000", 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014495600000000001 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:839 3.es0 3:7773fe95:::smithi19125844-37:head [setxattr _header (57) in=7b] snapc 2f=[2f,2e,2d,2b,28,26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:50.485 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.782914181999999, 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.036686074999999999, 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 839 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:50.486 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605814+0000", 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605815+0000", 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1909999999999999e-06 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.487 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605822+0000", 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3369999999999996e-06 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605840+0000", 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7949e-05 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605848+0000", 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6980000000000004e-06 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.488 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631010+0000", 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025161701000000002 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631081+0000", 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0993000000000003e-05 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631379+0000", 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000298146 2024-09-15T02:49:50.489 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.632106+0000", 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072663800000000005 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642331+0000", 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010225316 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.642500+0000", 2024-09-15T02:49:50.490 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000169106 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:840 3.es0 3:7773fe95:::smithi19125844-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 22.782870558999999, 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.039020892000000001, 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 840 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605858+0000", 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605859+0000", 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.02e-06 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605864+0000", 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1000000000000003e-06 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605882+0000", 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8128999999999999e-05 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.493 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.605891+0000", 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.5550000000000003e-06 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631895+0000", 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026004328 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.631958+0000", 2024-09-15T02:49:50.494 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2603999999999999e-05 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643304+0000", 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01134697 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.643450+0000", 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000145069 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.495 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:27.644879+0000", 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0014291169999999999 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.496 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:50.497 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:50.498 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:50.496+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:50.598 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:50.596+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:50.604 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:50.604 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:50.604 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:50.604 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:50.614 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:50.700 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:50.698+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:50.713 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:50.713 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:50.713 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:50.723 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:50.800 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:50.800+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/33 ECSubWriteReply(tid=38, last_complete=44'25, committed=1, applied=1))", 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 62.206023518999999, 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72693023700000003, 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618855+0000", 2024-09-15T02:49:50.830 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618856+0000", 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0380000000000001e-06 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:48.618863+0000", 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2429999999999999e-06 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.831 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343770+0000", 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.72490705799999999 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343784+0000", 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3726e-05 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.832 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.343918+0000", 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000133781 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:49.345785+0000", 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0018673909999999999 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:416 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~638976] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:50.833 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.563815030000001, 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.026381341999999999, 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 416 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261063+0000", 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.261066+0000", 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1159999999999999e-06 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262194+0000", 2024-09-15T02:49:50.835 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001127783 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262245+0000", 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0880000000000001e-05 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262260+0000", 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5065999999999999e-05 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.836 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262382+0000", 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012151599999999999 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.262403+0000", 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1344000000000001e-05 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266706+0000", 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.004303355 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.837 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.266794+0000", 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7717000000000001e-05 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.268260+0000", 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001466406 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285160+0000", 2024-09-15T02:49:50.838 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016899773 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287372+0000", 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022116140000000002 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287445+0000", 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2772000000000001e-05 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.839 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:417 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~622592] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.554421079999997, 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.028569959999999998, 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.840 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 417 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270457+0000", 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.841 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.270459+0000", 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.035e-06 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271813+0000", 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001354037 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271834+0000", 2024-09-15T02:49:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0302e-05 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271844+0000", 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9070000000000002e-06 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274939+0000", 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.003095169 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.843 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.275041+0000", 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000102045 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.276123+0000", 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010819880000000001 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285162+0000", 2024-09-15T02:49:50.844 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009039432 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.298984+0000", 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013821599 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299027+0000", 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.3445999999999997e-05 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.845 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:418 3.0s0 3:038350ba:::smithi19125844-11:head [append 0~147456] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.553018579000003, 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027304571999999999, 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 418 2024-09-15T02:49:50.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.847 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271860+0000", 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.271861+0000", 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1829999999999999e-06 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272037+0000", 2024-09-15T02:49:50.848 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017563799999999999 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272050+0000", 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3506000000000001e-05 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272057+0000", 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2799999999999998e-06 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.849 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281725+0000", 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096680910000000002 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.281887+0000", 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016169599999999999 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.850 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.282772+0000", 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00088466100000000002 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285439+0000", 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0026669340000000001 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299113+0000", 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013674009000000001 2024-09-15T02:49:50.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.299164+0000", 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1573999999999998e-05 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:419 3.0s0 3:038350ba:::smithi19125844-11:head [setxattr _header (56) in=7b] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.552813282000002, 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.031729234000000002, 2024-09-15T02:49:50.852 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 419 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.853 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272065+0000", 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272066+0000", 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9000000000000005e-07 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272070+0000", 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1740000000000002e-06 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.854 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272078+0000", 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5329999999999997e-06 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272083+0000", 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7859999999999996e-06 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.855 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.856 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284520+0000", 2024-09-15T02:49:50.856 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012437190000000001 2024-09-15T02:49:50.856 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284611+0000", 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0816000000000006e-05 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.284926+0000", 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031568500000000001 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.857 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.287991+0000", 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0030649599999999998 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303704+0000", 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015713142999999999 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.303794+0000", 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9957000000000006e-05 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:420 3.0s0 3:038350ba:::smithi19125844-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.552790275, 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.033404771999999999, 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 420 2024-09-15T02:49:50.860 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.860 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.860 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.860 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.860 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:50.860 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272088+0000", 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272089+0000", 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.51e-07 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.861 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.862 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272091+0000", 2024-09-15T02:49:50.862 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1730000000000002e-06 2024-09-15T02:49:50.862 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.863 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.863 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.863 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272096+0000", 2024-09-15T02:49:50.864 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.904e-06 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272099+0000", 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2059999999999999e-06 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285286+0000", 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013187488000000001 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.865 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.285374+0000", 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.7392999999999995e-05 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304237+0000", 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018863347999999999 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.304386+0000", 2024-09-15T02:49:50.866 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014841 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.305493+0000", 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001107299 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:429 3.9s0 3:99e165cc:::smithi19125844-19:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.552716928000002, 2024-09-15T02:49:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022788739999999998, 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 429 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:50.868 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272161+0000", 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272162+0000", 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7400000000000003e-07 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.869 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272166+0000", 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.9269999999999998e-06 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272174+0000", 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6569999999999997e-06 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.870 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.871 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.272178+0000", 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6260000000000003e-06 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274127+0000", 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001949295 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.274178+0000", 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0547999999999998e-05 2024-09-15T02:49:50.872 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.294950+0000", 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.020772112999999998 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpReadReply(3.9s0 50/33 ECSubReadReply(tid=60, attrs_read=0))", 2024-09-15T02:49:50.873 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 56.547443659999999, 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017580023, 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:50.874 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.277435+0000", 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0100000000000004e-07 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293753+0000", 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016317505999999999 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.875 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293859+0000", 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000105707 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293862+0000", 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.4819999999999999e-06 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.876 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.293910+0000", 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.7763000000000003e-05 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:54.295015+0000", 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0011048640000000001 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.877 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.878 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.878 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(osd.12.22:2 3.9s0 3:94f52c52:::smithi19125844-20:head [assert-version v18,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-15T02:49:50.878 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:50.878 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 51.497022008999998, 2024-09-15T02:49:50.878 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59485876100000001, 2024-09-15T02:49:50.878 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327856+0000", 2024-09-15T02:49:50.879 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327859+0000", 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.446e-06 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327869+0000", 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7629999999999995e-06 2024-09-15T02:49:50.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327898+0000", 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9054e-05 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.327913+0000", 2024-09-15T02:49:50.881 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4990999999999999e-05 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328005+0000", 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2015000000000006e-05 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.328180+0000", 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017567599999999999 2024-09-15T02:49:50.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.922715+0000", 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.59453481600000002 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:630 3.9s0 3:9c49cf3d:::smithi19125844-14:head [copy-from ver 31] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.430967447, 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021438329999999998, 2024-09-15T02:49:50.883 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 630 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.884 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393911+0000", 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393915+0000", 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7570000000000002e-06 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393951+0000", 2024-09-15T02:49:50.885 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6347999999999998e-05 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.393987+0000", 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6405999999999997e-05 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394003+0000", 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5956000000000001e-05 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394131+0000", 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012749300000000001 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394155+0000", 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3754999999999999e-05 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401719+0000", 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0075645340000000004 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.401977+0000", 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000257543 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.409969+0000", 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007991873 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.414563+0000", 2024-09-15T02:49:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0045945830000000002 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "commit_sent", 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415199+0000", 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00063606199999999996 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.415349+0000", 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015002 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:631 3.9s0 3:9c49cf3d:::smithi19125844-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 38.430826508999999, 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017944699000000001, 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 631 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394052+0000", 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394053+0000", 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3179999999999999e-06 2024-09-15T02:49:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394059+0000", 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9120000000000003e-06 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394075+0000", 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5809999999999999e-05 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394085+0000", 2024-09-15T02:49:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0586e-05 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394258+0000", 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017270999999999999 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.394277+0000", 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9236999999999999e-05 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402333+0000", 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0080553299999999994 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting for blocked object", 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.402447+0000", 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00011408600000000001 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411771+0000", 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0093238820000000004 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411860+0000", 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8857999999999995e-05 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:12.411997+0000", 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00013697000000000001 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 72/71 ECSubWriteReply(tid=134, last_complete=72'51, committed=1, applied=1))", 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 34.532036476000002, 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060209891000000001, 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292842+0000", 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292843+0000", 2024-09-15T02:49:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.3099999999999996e-07 2024-09-15T02:49:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.292848+0000", 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6729999999999997e-06 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352898+0000", 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.060049335000000002 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352910+0000", 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.256e-05 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.352988+0000", 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7831999999999996e-05 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.353052+0000", 2024-09-15T02:49:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3559999999999995e-05 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:688 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.386005462999996, 2024-09-15T02:49:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037318727000000003, 2024-09-15T02:49:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 688 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438873+0000", 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438875+0000", 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5809999999999999e-06 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438889+0000", 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3444999999999999e-05 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438927+0000", 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.7842000000000001e-05 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438940+0000", 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3473e-05 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439086+0000", 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014553999999999999 2024-09-15T02:49:50.907 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "op must wait for map", 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439109+0000", 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3283000000000001e-05 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448063+0000", 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0089537490000000004 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.908 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448274+0000", 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000210984 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476192+0000", 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027917830000000001 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:689 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:50.909 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.385916475999998, 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037724385999999999, 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 689 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.910 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438962+0000", 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438963+0000", 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5300000000000002e-07 2024-09-15T02:49:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438968+0000", 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9180000000000002e-06 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438976+0000", 2024-09-15T02:49:50.912 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.6470000000000004e-06 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.438981+0000", 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9130000000000002e-06 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439306+0000", 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00032486700000000001 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:50.913 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439319+0000", 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2442999999999999e-05 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.448872+0000", 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095532159999999998 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449323+0000", 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00045103599999999998 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.914 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.476686+0000", 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027363393 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "osd_op(client.4526.0:690 3.0s0 3:038350ba:::smithi19125844-11:b [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 32.385834326000001, 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038145563, 2024-09-15T02:49:50.915 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "client_info": { 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "client": "client.4526", 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "tid": 690 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.916 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439044+0000", 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439045+0000", 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2200000000000003e-07 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439048+0000", 2024-09-15T02:49:50.917 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.76e-06 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439064+0000", 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5336e-05 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439069+0000", 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.8849999999999998e-06 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.918 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439382+0000", 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00031364799999999998 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "waiting_for_map not empty", 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.439397+0000", 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4661000000000001e-05 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449673+0000", 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010276298 2024-09-15T02:49:50.919 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.449816+0000", 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014270799999999999 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:18.477190+0000", 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.027373544999999999 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.920 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDScrubReserve(3.19s4 RELEASE e75)", 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 30.542843435999998, 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98297010699999998, 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "started", 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.921 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282035+0000", 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282036+0000", 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2270000000000001e-06 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.922 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:20.282044+0000", 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6720000000000004e-06 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264750+0000", 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.98270659599999999 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264763+0000", 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2466000000000001e-05 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.923 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264912+0000", 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000149524 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "started", 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.264954+0000", 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1461000000000002e-05 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.265005+0000", 2024-09-15T02:49:50.924 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.1161e-05 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.8s4 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 29.290565909000001, 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.018239887999999999, 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.925 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534312+0000", 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534314+0000", 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7549999999999999e-06 2024-09-15T02:49:50.926 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.534957+0000", 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00064235200000000001 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535041+0000", 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4050999999999999e-05 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535056+0000", 2024-09-15T02:49:50.927 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5588999999999999e-05 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535177+0000", 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000120925 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.535207+0000", 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9864000000000001e-05 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.928 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552499+0000", 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017291890000000001 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.552552+0000", 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.3461999999999997e-05 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.bs4 77/76 [PushOp(3:d394ce14:::smithi19125844-2:head, version: 71'77, data_included: [262144~241664], data_size: 241664, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d394ce14:::smithi19125844-2:head@71'77, size: 2015232, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2015232, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:50.929 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 29.188432495000001, 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.24636945099999999, 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636446+0000", 2024-09-15T02:49:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636447+0000", 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.637e-06 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.636697+0000", 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00024920600000000003 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.931 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867157+0000", 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.23046036 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867176+0000", 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.929e-05 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.867332+0000", 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015594099999999999 2024-09-15T02:49:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:21.882815+0000", 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.015483017 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:50.933 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 28.327391933000001, 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025018460999999999, 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497486+0000", 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.934 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497487+0000", 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8000000000000002e-07 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499654+0000", 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00216738 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499711+0000", 2024-09-15T02:49:50.935 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.7398999999999997e-05 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499723+0000", 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2019e-05 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499830+0000", 2024-09-15T02:49:50.936 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000106718 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499850+0000", 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9732e-05 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522437+0000", 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022587494 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.937 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.522505+0000", 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.7439000000000003e-05 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs4 93/76 ECSubWrite(tid=156, reqid=client.4526.0:952, at_version=93'25, trim_to=0'0, roll_forward_to=85'23))", 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 13.538741701999999, 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.038822470999999997, 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:50.938 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.286137+0000", 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.4300000000000003e-07 2024-09-15T02:49:50.939 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292226+0000", 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0060890609999999998 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292292+0000", 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6272000000000005e-05 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292303+0000", 2024-09-15T02:49:50.940 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0111e-05 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292409+0000", 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00010671 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.292423+0000", 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3424999999999999e-05 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:50.941 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.324890+0000", 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.032467754000000001 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.324959+0000", 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8595000000000003e-05 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:50.942 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:50.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:50.945 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:50.944+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:51.000 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:51.032 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:51.035 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:51.035 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:51.035 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:51.036 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:51.045 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:51.042+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:51.056 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.058 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:51.058 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:51.058 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:51.146 INFO:tasks.ceph.osd.6.smithi152.stderr:2024-09-15T02:49:51.142+0000 7fd7bf595700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34472) UID: 0 2024-09-15T02:49:51.168 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.168 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:51.168 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:51.169 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:51.169 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:51.247 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:51.244+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 67.703531690999995, 2024-09-15T02:49:51.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019828684999999999, 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569442+0000", 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9999999999999999e-07 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570188+0000", 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074560499999999999 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570236+0000", 2024-09-15T02:49:51.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8504999999999997e-05 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570244+0000", 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.994e-06 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570483+0000", 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000238929 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570510+0000", 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7083000000000001e-05 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589246+0000", 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018736292000000002 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.589270+0000", 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4077e-05 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 60.622585008999998, 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015141226000000001, 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650388+0000", 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650389+0000", 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8000000000000001e-07 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650600+0000", 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021155999999999999 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650679+0000", 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8641999999999997e-05 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650693+0000", 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4294000000000001e-05 2024-09-15T02:49:51.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650801+0000", 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108132 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.650828+0000", 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6565999999999999e-05 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665503+0000", 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014675142 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.665530+0000", 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6509999999999999e-05 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:51.285 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.580490775000001, 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011398508, 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692483+0000", 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.22e-07 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693204+0000", 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072156 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693258+0000", 2024-09-15T02:49:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3134999999999999e-05 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693266+0000", 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5259999999999993e-06 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693333+0000", 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6580000000000003e-05 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.693344+0000", 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0971999999999999e-05 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703854+0000", 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010510265 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.703881+0000", 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7248000000000001e-05 2024-09-15T02:49:51.289 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.018660146000002, 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016572159999999999, 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254313+0000", 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254314+0000", 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1100000000000002e-07 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255086+0000", 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077269100000000005 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255147+0000", 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1005999999999998e-05 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255166+0000", 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8581999999999999e-05 2024-09-15T02:49:51.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255267+0000", 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100973 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255289+0000", 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2565000000000002e-05 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270855+0000", 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015565733999999999 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.270885+0000", 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0298e-05 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.0s3 50/33 ECSubWrite(tid=57, reqid=client.4526.0:416, at_version=50'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:51.294 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.004169191000003, 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018797873, 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.268804+0000", 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8999999999999998e-07 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269160+0000", 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035577800000000001 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269204+0000", 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3349000000000001e-05 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269211+0000", 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9439999999999999e-06 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269283+0000", 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2263999999999998e-05 2024-09-15T02:49:51.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.269296+0000", 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3145e-05 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287519+0000", 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018222955999999998 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.287602+0000", 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3146999999999995e-05 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(3.1s2 RELEASE e76)", 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.603338652000001, 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43008043499999998, 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:51.299 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669635+0000", 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669637+0000", 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.917e-06 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:20.669648+0000", 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1653e-05 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099516+0000", 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42986811200000002 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099534+0000", 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7997000000000001e-05 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099653+0000", 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011822100000000001 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099681+0000", 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8496999999999998e-05 2024-09-15T02:49:51.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.099715+0000", 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4038000000000002e-05 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:head, version: 72'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:head@72'42, size: 2392064, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.789247668000002, 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012988501, 2024-09-15T02:49:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483726+0000", 2024-09-15T02:49:51.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483727+0000", 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2500000000000004e-07 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483886+0000", 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000159028 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483939+0000", 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3640999999999997e-05 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.483950+0000", 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0462000000000001e-05 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.484771+0000", 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082086099999999999 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.496714+0000", 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011943684 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:127fa005:::smithi19125844-18:head, version: 36'10, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:127fa005:::smithi19125844-18:head@36'10, size: 1212416, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.763179379, 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012919893999999999, 2024-09-15T02:49:51.307 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509794+0000", 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.509795+0000", 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8599999999999996e-07 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510077+0000", 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028238699999999999 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510142+0000", 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4518999999999997e-05 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.510161+0000", 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8751999999999999e-05 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.511930+0000", 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017696719999999999 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.522714+0000", 2024-09-15T02:49:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010783578 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=250, reqid=client.4526.0:760, at_version=77'44, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.738404883000001, 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0218164, 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534569+0000", 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6199999999999996e-07 2024-09-15T02:49:51.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534737+0000", 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016761400000000001 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534787+0000", 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9836000000000002e-05 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534795+0000", 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1550000000000007e-06 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534859+0000", 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4162e-05 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.534875+0000", 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6305000000000001e-05 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556345+0000", 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021469637999999999 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556385+0000", 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9827999999999999e-05 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.8s6 77/76 ECSubWrite(tid=251, reqid=client.4526.0:761, at_version=77'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:51.315 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.730610117000001, 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014142234, 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542363+0000", 2024-09-15T02:49:51.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542364+0000", 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8700000000000005e-07 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542468+0000", 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010437099999999999 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542522+0000", 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3295999999999999e-05 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.542529+0000", 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1709999999999996e-06 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544564+0000", 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00203476 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.544579+0000", 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5605000000000001e-05 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556487+0000", 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011907687 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.556506+0000", 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556999999999999e-05 2024-09-15T02:49:51.319 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.8s6 77/76 [PushOp(3:173bfaf3:::smithi19125844-44:18, version: 70'39, data_included: [262144~253952], data_size: 253952, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:173bfaf3:::smithi19125844-44:18@70'39, size: 2064384, copy_subset: [], clone_subset: {}, snapset: 27=[]:{e=[b],18=[18],27=[27,26,25,24,21,1d]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2064384, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.57475123, 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010750387, 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698222+0000", 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698224+0000", 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2570000000000002e-06 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698584+0000", 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035969000000000002 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698700+0000", 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116018 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698723+0000", 2024-09-15T02:49:51.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3173e-05 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.698830+0000", 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010714599999999999 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.708973+0000", 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010142103 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.as3 81/76 ECSubRead(tid=114, to_read={3:517ce0e6:::smithi19125844-47:head=0,475136,0}, subchunks={3:517ce0e6:::smithi19125844-47:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.783863722, 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69833002899999996, 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489110+0000", 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489111+0000", 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7269999999999999e-06 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:25.489134+0000", 2024-09-15T02:49:51.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3033000000000001e-05 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187052+0000", 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69791777600000005 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187066+0000", 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3567000000000001e-05 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187169+0000", 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010303599999999999 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:26.187440+0000", 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027088999999999998 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "replica_scrub(pg: 3.0s3,from:0'0,to:85'34,epoch:86/71,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:51.327 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.855523311999999, 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98343883300000001, 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:51.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417450+0000", 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9999999999999999e-07 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:30.417455+0000", 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-06 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400723+0000", 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98326817399999999 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400733+0000", 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5929999999999995e-06 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400839+0000", 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106413 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400855+0000", 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5150000000000001e-05 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:31.400889+0000", 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4353000000000001e-05 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=173, reqid=client.4526.0:901, at_version=88'59, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.992138651000001, 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019105225999999999, 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280835+0000", 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7999999999999996e-07 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281129+0000", 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029418700000000001 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281252+0000", 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123051 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281268+0000", 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5682999999999999e-05 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281408+0000", 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013955900000000001 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281431+0000", 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2952e-05 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299857+0000", 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018425891 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.299940+0000", 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3423000000000004e-05 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=175, reqid=client.4526.0:903, at_version=88'61, trim_to=0'0, roll_forward_to=85'57))", 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.97084723, 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011439048, 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302126+0000", 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302127+0000", 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.119e-06 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302328+0000", 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000200834 2024-09-15T02:49:51.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302386+0000", 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7895e-05 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302394+0000", 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0770000000000008e-06 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302465+0000", 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1017000000000001e-05 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302481+0000", 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5599000000000001e-05 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313526+0000", 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011045743 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313565+0000", 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8763999999999997e-05 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.15s6 88/76 ECSubWrite(tid=178, reqid=client.4526.0:905, at_version=0'0, trim_to=0'0, roll_forward_to=88'63))", 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 18.953231873, 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15873490300000001, 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.341 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319742+0000", 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319766+0000", 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3816e-05 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476980+0000", 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.157214144 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.476993+0000", 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2286000000000001e-05 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477134+0000", 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141006 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.477167+0000", 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3847000000000003e-05 2024-09-15T02:49:51.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478412+0000", 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012443930000000001 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.478476+0000", 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4579999999999995e-05 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.19s3 95/73 ECSubWrite(tid=23, reqid=osd.9.0:23, at_version=95'65, trim_to=0'0, roll_forward_to=93'63))", 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.021443177, 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01234181, 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251530+0000", 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.29e-07 2024-09-15T02:49:51.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251554+0000", 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4042000000000001e-05 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251590+0000", 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5329e-05 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.251596+0000", 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7130000000000002e-06 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.256779+0000", 2024-09-15T02:49:51.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051832880000000003 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.256788+0000", 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0890000000000006e-06 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263842+0000", 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070541850000000001 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263872+0000", 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9935000000000001e-05 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 95/71 ECSubWrite(tid=539, reqid=client.4526.0:958, at_version=95'82, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.002599629000001, 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025699248000000001, 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.351 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270374+0000", 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.73e-07 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270803+0000", 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042948800000000001 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270838+0000", 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4785e-05 2024-09-15T02:49:51.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270851+0000", 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2347000000000001e-05 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270899+0000", 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8164999999999997e-05 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270909+0000", 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0614e-05 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.295820+0000", 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024910394999999998 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296073+0000", 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000253281 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.355 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 95/71 ECSubWrite(tid=542, reqid=client.4526.0:959, at_version=95'83, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.278951+0000", 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.994022734, 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017151198999999999, 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.278951+0000", 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.278951+0000", 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.278952+0000", 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1260000000000001e-06 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279680+0000", 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072793700000000001 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279725+0000", 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4894000000000001e-05 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279732+0000", 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-06 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.280070+0000", 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033776699999999999 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.280080+0000", 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0345e-05 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296083+0000", 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016002407999999999 2024-09-15T02:49:51.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296102+0000", 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9171999999999999e-05 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s2 95/71 ECSubWrite(tid=544, reqid=client.4526.0:960, at_version=95'84, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.286980+0000", 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.985993431000001, 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011104908, 2024-09-15T02:49:51.360 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.286980+0000", 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.286980+0000", 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.286981+0000", 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.47e-07 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.287714+0000", 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00073319499999999996 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.287752+0000", 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8025000000000003e-05 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.287760+0000", 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7559999999999999e-06 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.289482+0000", 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017221350000000001 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.289492+0000", 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9690000000000006e-06 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.297996+0000", 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085040859999999992 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.298085+0000", 2024-09-15T02:49:51.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8795000000000006e-05 2024-09-15T02:49:51.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.365 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.365 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.365 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:51.367 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:51.364+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:51.467 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:51.462+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:51.514 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.515 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:51.515 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:51.515 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:51.568 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:51.563+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:51.625 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.625 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:51.625 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:51.625 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:51.625 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:51.655 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14, 15, 10] out_osds: [6] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 14, 3, 7, 11, 15, 10, 6] 2024-09-15T02:49:51.655 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T02:49:51.655 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-15T02:49:51.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-09-15T02:49:51.669 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:51.666+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=90, reqid=client.4526.0:221, at_version=39'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.166896557000001, 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024151102000000001, 2024-09-15T02:49:51.740 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569741+0000", 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.569743+0000", 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-06 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570676+0000", 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093292699999999995 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570768+0000", 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2439e-05 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570786+0000", 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7986999999999999e-05 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570883+0000", 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6490000000000006e-05 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.570901+0000", 2024-09-15T02:49:51.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8858000000000001e-05 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593864+0000", 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022962737 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593892+0000", 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7844e-05 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=93, reqid=client.4526.0:222, at_version=39'11, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.154380372000006, 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011923367000000001, 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582257+0000", 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582258+0000", 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2699999999999998e-07 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582392+0000", 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013381800000000001 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582448+0000", 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5986000000000002e-05 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.582457+0000", 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9570000000000008e-06 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590801+0000", 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083444299999999999 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.590811+0000", 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9129999999999998e-06 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594134+0000", 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033230909999999998 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594180+0000", 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6245000000000002e-05 2024-09-15T02:49:51.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 39/33 ECSubWrite(tid=95, reqid=client.4526.0:223, at_version=39'12, trim_to=0'0, roll_forward_to=36'8))", 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.153341212000001, 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011268293, 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583296+0000", 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583297+0000", 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583323+0000", 2024-09-15T02:49:51.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6176000000000001e-05 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583479+0000", 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155346 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.583488+0000", 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.952e-06 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593691+0000", 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01020305 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.593705+0000", 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4649e-05 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594549+0000", 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084332799999999996 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.594565+0000", 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5943999999999998e-05 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/33 ECSubWrite(tid=163, reqid=client.4526.0:312, at_version=46'23, trim_to=0'0, roll_forward_to=44'21))", 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 61.085590205999999, 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015031548, 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651047+0000", 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651048+0000", 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4900000000000001e-07 2024-09-15T02:49:51.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651266+0000", 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000218233 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651324+0000", 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8544000000000001e-05 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651335+0000", 2024-09-15T02:49:51.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336000000000001e-05 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651408+0000", 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3415000000000001e-05 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.651424+0000", 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6001000000000001e-05 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666045+0000", 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014620861000000001 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:50.666079+0000", 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3809000000000001e-05 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/33 ECSubWrite(tid=158, reqid=client.4526.0:363, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:51.758 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.044217822, 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015570855999999999, 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692420+0000", 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3799999999999996e-07 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692736+0000", 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031561799999999999 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692809+0000", 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3174000000000001e-05 2024-09-15T02:49:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692821+0000", 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1627000000000001e-05 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692906+0000", 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5172000000000004e-05 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.692920+0000", 2024-09-15T02:49:51.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4361e-05 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707954+0000", 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015033451999999999 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.707991+0000", 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6714000000000001e-05 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/33 ECSubWrite(tid=176, reqid=client.4526.0:384, at_version=49'52, trim_to=0'0, roll_forward_to=46'50))", 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.987155667000003, 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014530774999999999, 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749482+0000", 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749483+0000", 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4799999999999997e-07 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749717+0000", 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000234408 2024-09-15T02:49:51.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749768+0000", 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1155000000000001e-05 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749777+0000", 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1609999999999992e-06 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749853+0000", 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5803000000000006e-05 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.749872+0000", 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8556e-05 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.763971+0000", 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014098934 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.764013+0000", 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1810000000000001e-05 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 50/33 ECSubWrite(tid=190, reqid=client.4526.0:409, at_version=50'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-15T02:49:51.767 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.482275241000004, 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012056918, 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254362+0000", 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254363+0000", 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7199999999999997e-07 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.254896+0000", 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053267899999999999 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255034+0000", 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013821100000000001 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255066+0000", 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2128000000000001e-05 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255135+0000", 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8885000000000002e-05 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255148+0000", 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2498e-05 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266383+0000", 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011235187000000001 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.266419+0000", 2024-09-15T02:49:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6358e-05 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=197, reqid=client.4526.0:430, at_version=50'59, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.445500635999998, 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011905962000000001, 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291137+0000", 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291138+0000", 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.31e-06 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291342+0000", 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203403 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291572+0000", 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000230084 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291579+0000", 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6990000000000003e-06 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291641+0000", 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2093000000000003e-05 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291650+0000", 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1759999999999993e-06 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303018+0000", 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011368296 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303043+0000", 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4901000000000001e-05 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.776 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.7s2 50/33 ECSubWrite(tid=201, reqid=client.4526.0:431, at_version=50'60, trim_to=0'0, roll_forward_to=49'57))", 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.445266246000003, 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011692335, 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291371+0000", 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9500000000000001e-07 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291548+0000", 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000177021 2024-09-15T02:49:51.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291673+0000", 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012469299999999999 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.291678+0000", 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4710000000000001e-06 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296939+0000", 2024-09-15T02:49:51.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052609120000000004 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.296948+0000", 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303051+0000", 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006102433 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.303064+0000", 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2948999999999999e-05 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/33 ECSubWrite(tid=212, reqid=client.4526.0:450, at_version=52'22, trim_to=0'0, roll_forward_to=49'21))", 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.488178062999999, 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015878961, 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248459+0000", 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.248460+0000", 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249338+0000", 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087768999999999998 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249433+0000", 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5333000000000004e-05 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249451+0000", 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8406e-05 2024-09-15T02:49:51.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249656+0000", 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020463100000000001 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.249685+0000", 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8943000000000001e-05 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264308+0000", 2024-09-15T02:49:51.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014623486999999999 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:56.264338+0000", 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0020999999999999e-05 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=404, reqid=client.4526.0:671, at_version=72'78, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 35.449980316999998, 2024-09-15T02:49:51.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020897651, 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286657+0000", 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.286658+0000", 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.58e-07 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287075+0000", 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041690100000000001 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287318+0000", 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024316499999999999 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287324+0000", 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9580000000000004e-06 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287521+0000", 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197746 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287532+0000", 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366e-05 2024-09-15T02:49:51.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307520+0000", 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019987953999999999 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307555+0000", 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5102999999999999e-05 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.789 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=407, reqid=client.4526.0:672, at_version=72'79, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 35.449543564999999, 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020477789999999999, 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287094+0000", 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3200000000000001e-07 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287283+0000", 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018875200000000001 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287367+0000", 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4036000000000007e-05 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.287374+0000", 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5340000000000002e-06 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295137+0000", 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077628599999999999 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.295149+0000", 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307560+0000", 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012411447000000001 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307572+0000", 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.793 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 72/33 ECSubWrite(tid=413, reqid=client.4526.0:673, at_version=72'80, trim_to=0'0, roll_forward_to=71'77))", 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 35.443700737999997, 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015088048, 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292937+0000", 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8799999999999999e-07 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.292992+0000", 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5479000000000002e-05 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293291+0000", 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029850500000000002 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.293295+0000", 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4179999999999999e-06 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302806+0000", 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095105680000000005 2024-09-15T02:49:51.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.302817+0000", 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1464999999999999e-05 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.307993+0000", 2024-09-15T02:49:51.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051757540000000003 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.308025+0000", 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1671000000000002e-05 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.798 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:c, version: 71'82, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:c@71'82, size: 1081344, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.021952046999999, 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013023088, 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714685+0000", 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.714687+0000", 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.866e-06 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715107+0000", 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419293 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715195+0000", 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8758999999999995e-05 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715215+0000", 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0143e-05 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.715370+0000", 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000154611 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.727709+0000", 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012338416 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7773fe95:::smithi19125844-37:head, version: 71'83, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7773fe95:::smithi19125844-37:head@71'83, size: 2293760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.751115157000001, 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011824942, 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.802 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985522+0000", 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985524+0000", 2024-09-15T02:49:51.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.533e-06 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985735+0000", 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021152599999999999 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985816+0000", 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0516e-05 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985833+0000", 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7413000000000001e-05 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.985974+0000", 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141161 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.997347+0000", 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011372793000000001 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.805 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(3.es2 77/76 [PushOp(3:7f1f21a6:::smithi19125844-10:head, version: 71'85, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:7f1f21a6:::smithi19125844-10:head@71'85, size: 1736704, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.492018608999999, 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014391049, 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:51.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244619+0000", 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.244620+0000", 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4300000000000001e-06 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245318+0000", 2024-09-15T02:49:51.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069816099999999999 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245404+0000", 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5782e-05 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245423+0000", 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8272999999999999e-05 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.245544+0000", 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121785 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.259010+0000", 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013465618 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=472, reqid=client.4526.0:793, at_version=79'84, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.25317042, 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013978862999999999, 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483467+0000", 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483468+0000", 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1119999999999999e-06 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483595+0000", 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012675800000000001 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483644+0000", 2024-09-15T02:49:51.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9200000000000003e-05 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483653+0000", 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4030000000000007e-06 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483747+0000", 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4239000000000004e-05 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.483762+0000", 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5698000000000001e-05 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497313+0000", 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013550859 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.497446+0000", 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013259400000000001 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.bs6 79/76 ECSubWrite(tid=473, reqid=client.4526.0:794, at_version=79'85, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.246955191000001, 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014861180999999999, 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489682+0000", 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.489684+0000", 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.282e-06 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490105+0000", 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042170800000000002 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490163+0000", 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8158000000000001e-05 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490173+0000", 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4350000000000003e-06 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490301+0000", 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127928 2024-09-15T02:49:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.490314+0000", 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3104e-05 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504495+0000", 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014181545 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:23.504543+0000", 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8021e-05 2024-09-15T02:49:51.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 95/71 ECSubWrite(tid=539, reqid=client.4526.0:958, at_version=95'82, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.465979484, 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025430995000000001, 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:51.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270658+0000", 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.270659+0000", 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271073+0000", 2024-09-15T02:49:51.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000414342 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271106+0000", 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3107e-05 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271113+0000", 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5910000000000001e-06 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271176+0000", 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3633999999999994e-05 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.271187+0000", 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0436e-05 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296049+0000", 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024862042000000001 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296089+0000", 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0102000000000003e-05 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.1s6 95/71 ECSubWrite(tid=542, reqid=client.4526.0:959, at_version=95'83, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.45753597, 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017183397, 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:51.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279102+0000", 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279111+0000", 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.251e-06 2024-09-15T02:49:51.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279556+0000", 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044511799999999999 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279593+0000", 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.663e-05 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279600+0000", 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1400000000000002e-06 2024-09-15T02:49:51.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279671+0000", 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.161e-05 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.279681+0000", 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7559999999999994e-06 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296269+0000", 2024-09-15T02:49:51.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016588354 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.296285+0000", 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5537999999999999e-05 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:51.826 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:51.828 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:51.824+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:51.929 INFO:tasks.ceph.osd.10.smithi152.stderr:2024-09-15T02:49:51.926+0000 7f893ea5b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 27180) UID: 0 2024-09-15T02:49:51.966 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-15T02:49:51.959+0000 7f08e39de700 -1 mon.a@0(leader).osd e105 definitely_dead 0 2024-09-15T02:49:51.974 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:51.974 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:51.974 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:51.974 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:51.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:52.030 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:52.028+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:52.082 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:52.082 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:52.082 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:52.082 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:52.082 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:52.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:52.131 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:52.127+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:52.231 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:52.230+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.868247214999997, 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.046578420000000002, 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363746+0000", 2024-09-15T02:49:52.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363748+0000", 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4370000000000001e-06 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363763+0000", 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5160000000000001e-05 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363789+0000", 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6227000000000001e-05 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363804+0000", 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4884e-05 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363921+0000", 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116753 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.363935+0000", 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3526999999999999e-05 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407540+0000", 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043605572000000002 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.407860+0000", 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032016800000000002 2024-09-15T02:49:52.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.408121+0000", 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026108799999999998 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.409358+0000", 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001236699 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410165+0000", 2024-09-15T02:49:52.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080705399999999997 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410262+0000", 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7283999999999999e-05 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.410324+0000", 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1567000000000002e-05 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.508675701000001, 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42703490700000002, 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 52 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723317+0000", 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723318+0000", 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0299999999999997e-07 2024-09-15T02:49:52.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723327+0000", 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7290000000000006e-06 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723350+0000", 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3419999999999999e-05 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.723359+0000", 2024-09-15T02:49:52.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1109999999999997e-06 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736670+0000", 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013310313000000001 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.736854+0000", 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184289 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:52.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.737070+0000", 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021579800000000001 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:24.740434+0000", 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003364513 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150139+0000", 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40970485299999998 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150232+0000", 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2544e-05 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.150352+0000", 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012043400000000001 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4189.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:52.246 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 86.663119257000005, 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58815473100000004, 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4189", 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3037228148", 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568874+0000", 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568875+0000", 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568880+0000", 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.463e-06 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568889+0000", 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7630000000000006e-06 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.568894+0000", 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581477+0000", 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012582164 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581543+0000", 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6249999999999998e-05 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2", 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.581641+0000", 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8369999999999995e-05 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.583658+0000", 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020171880000000001 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156866+0000", 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57320803300000001 2024-09-15T02:49:52.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156961+0000", 2024-09-15T02:49:52.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4783000000000004e-05 2024-09-15T02:49:52.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157028+0000", 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7187000000000006e-05 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.4189.0:100 1.0 e25/23 ondisk, result = 0)", 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 86.645600681000005, 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57065039799999995, 2024-09-15T02:49:52.253 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586392+0000", 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.43e-07 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:25.586394+0000", 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4079999999999999e-06 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156738+0000", 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57034380600000001 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156751+0000", 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2989999999999999e-05 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156840+0000", 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9137999999999999e-05 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.156853+0000", 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3283e-05 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:26.157043+0000", 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018963000000000001 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4311.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-15T02:49:52.257 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:52.259 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 84.831738254000001, 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95065160800000004, 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4311", 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.6:0/2554320040", 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:52.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400255+0000", 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400300+0000", 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5794000000000002e-05 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400330+0000", 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9855000000000001e-05 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400359+0000", 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8450999999999999e-05 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400475+0000", 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115889 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400923+0000", 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044863599999999998 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:27.400943+0000", 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9576999999999999e-05 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347485+0000", 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94654180399999999 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347734+0000", 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024947100000000002 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.347925+0000", 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190447 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.349919+0000", 2024-09-15T02:49:52.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019940549999999998 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350787+0000", 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086883499999999998 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350859+0000", 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1554e-05 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:28.350906+0000", 2024-09-15T02:49:52.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7240000000000002e-05 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:373 3.8s0 3:173bfaf3:::smithi19125844-44:head [append 0~98304] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.525494616000003, 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22554416399999999, 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.266 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 373 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706498+0000", 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706499+0000", 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2550000000000001e-06 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706538+0000", 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8197e-05 2024-09-15T02:49:52.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706547+0000", 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000005e-06 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706551+0000", 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718226+0000", 2024-09-15T02:49:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011674713999999999 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718306+0000", 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9673999999999996e-05 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.718758+0000", 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045167200000000002 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720562+0000", 2024-09-15T02:49:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018043499999999999 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.931829+0000", 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21126692799999999 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932042+0000", 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021359699999999999 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:374 3.8s0 3:173bfaf3:::smithi19125844-44:head [setxattr _header (56) in=7b] snapc e=[e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.525436783000004, 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225972861, 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 374 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706556+0000", 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706557+0000", 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-07 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706560+0000", 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5810000000000001e-06 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706580+0000", 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706586+0000", 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4060000000000002e-06 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719747+0000", 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013160583 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.719850+0000", 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103455 2024-09-15T02:49:52.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720252+0000", 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040231700000000002 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720927+0000", 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067460999999999997 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932323+0000", 2024-09-15T02:49:52.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21139606 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.932529+0000", 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020593799999999999 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:375 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.525401596999998, 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23538993899999999, 2024-09-15T02:49:52.277 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 375 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706591+0000", 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706592+0000", 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.06e-07 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706594+0000", 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4310000000000001e-06 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706603+0000", 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2770000000000005e-06 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.706606+0000", 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9079999999999997e-06 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720679+0000", 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014072214 2024-09-15T02:49:52.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.720780+0000", 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101063 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933567+0000", 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21278746300000001 2024-09-15T02:49:52.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.933825+0000", 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000258187 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:53.941981+0000", 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081557899999999996 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.6s4 54/53 ECSubWrite(tid=230, reqid=client.4526.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=54'12))", 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 53.790097037999999, 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84059119199999999, 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441896+0000", 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441897+0000", 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:58.441920+0000", 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2297e-05 2024-09-15T02:49:52.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280384+0000", 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83846432800000004 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280397+0000", 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3224000000000001e-05 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280743+0000", 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034603800000000001 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280768+0000", 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4275000000000001e-05 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282445+0000", 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001677542 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.282487+0000", 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1931999999999998e-05 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_update_log_missing(3.bs2 epoch 55/33 rep_tid 233 entries 55'26 (0'0) error 3:d4a23ffb:::smithi19125844-23:head by client.4526.0:467 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 52'24)", 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 52.951664825000002, 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033861255999999999, 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.287 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280328+0000", 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280329+0000", 2024-09-15T02:49:52.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4600000000000005e-07 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.280343+0000", 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4681999999999999e-05 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313053+0000", 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032709427999999999 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313069+0000", 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5982999999999999e-05 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.313240+0000", 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000171352 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:59.314189+0000", 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094926500000000003 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:670 3.8s0 3:173bfaf3:::smithi19125844-44:head [read 0~2392064,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e72)", 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 35.948370963000002, 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027236222000000001, 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 670 2024-09-15T02:49:52.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283622+0000", 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8800000000000003e-07 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283625+0000", 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3329999999999999e-06 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283630+0000", 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7450000000000001e-06 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283638+0000", 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7349999999999996e-06 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.283962+0000", 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032339899999999998 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.284074+0000", 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011208499999999999 2024-09-15T02:49:52.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298432+0000", 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014357939 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.298517+0000", 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5109000000000001e-05 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310858+0000", 2024-09-15T02:49:52.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012341388999999999 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpRead(3.bs2 72/33 ECSubRead(tid=420, to_read={3:d2661d0e:::smithi19125844-43:head=0,4096,0}, subchunks={3:d2661d0e:::smithi19125844-43:head=[0,1]}, attrs_to_read=))", 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 35.921189624, 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085265647999999999, 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310803+0000", 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310804+0000", 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.310814+0000", 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0115000000000001e-05 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395542+0000", 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.084728263999999998 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395552+0000", 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5899999999999997e-06 2024-09-15T02:49:52.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.395608+0000", 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5832999999999999e-05 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:16.396069+0000", 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046113399999999998 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:753 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.771241151000002, 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027324345, 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 753 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460752+0000", 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460754+0000", 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9709999999999998e-06 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460760+0000", 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7510000000000002e-06 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460773+0000", 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2136999999999999e-05 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460780+0000", 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9319999999999996e-06 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471867+0000", 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011087001000000001 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471873+0000", 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0710000000000002e-06 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.484980+0000", 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0131073370000001 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485369+0000", 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038873500000000002 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488076+0000", 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027074099999999999 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.304 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:754 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.771205562999999, 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0276021209999999, 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 754 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:52.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460787+0000", 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460788+0000", 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8899999999999999e-07 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460791+0000", 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.411e-06 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460798+0000", 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4679999999999997e-06 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460802+0000", 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2889999999999999e-06 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471896+0000", 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011093591 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471897+0000", 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.331e-06 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485874+0000", 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0139766539999999 2024-09-15T02:49:52.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.485962+0000", 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8481999999999998e-05 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488389+0000", 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024273060000000002 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:755 3.17s0 3:ee2f5e53:::smithi19125844-4:head [read 0~65536,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.771186545999999, 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0311918470000001, 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 755 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460806+0000", 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460807+0000", 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4999999999999998e-07 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460809+0000", 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460815+0000", 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1249999999999998e-06 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460819+0000", 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9670000000000003e-06 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471918+0000", 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098856000000001 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471919+0000", 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1200000000000002e-07 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488503+0000", 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0165840150000001 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488618+0000", 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011474199999999999 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491998+0000", 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033803650000000002 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.314 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:766 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.771169517000001, 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0356599440000001, 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 766 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460823+0000", 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460824+0000", 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.27e-07 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460831+0000", 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460837+0000", 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460841+0000", 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5889999999999999e-06 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471939+0000", 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011098591 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471940+0000", 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.488932+0000", 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0169917230000001 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.489075+0000", 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014318299999999999 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.496483+0000", 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074080589999999998 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.319 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:767 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.771148458999999, 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.037162342, 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 767 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460844+0000", 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460845+0000", 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3200000000000001e-07 2024-09-15T02:49:52.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460847+0000", 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.554e-06 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460853+0000", 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020000000000002e-06 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460857+0000", 2024-09-15T02:49:52.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0679999999999998e-06 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471961+0000", 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011103755999999999 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471962+0000", 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3600000000000003e-07 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.490940+0000", 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.018977662 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.491003+0000", 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3081000000000001e-05 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.498007+0000", 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070041510000000001 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.324 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:768 3.17s0 3:ec23d144:::smithi19125844-41:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.771131540999999, 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.038324754, 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.325 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 768 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460861+0000", 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460862+0000", 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.34e-07 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460864+0000", 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6639999999999998e-06 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460870+0000", 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0859999999999998e-06 2024-09-15T02:49:52.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.460874+0000", 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.563e-06 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471982+0000", 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011108122999999999 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-15T02:49:52.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:21.471983+0000", 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6700000000000003e-07 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492584+0000", 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0206010889999999 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.492705+0000", 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012115100000000001 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:22.499186+0000", 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006481077 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 95/77 ECSubWriteReply(tid=293, last_complete=95'119, committed=1, applied=1))", 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.968601405999999, 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55831691699999997, 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.330 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263391+0000", 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263392+0000", 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-09-15T02:49:52.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.263396+0000", 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0960000000000003e-06 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821430+0000", 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55803327199999997 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821437+0000", 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8320000000000006e-06 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821583+0000", 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014531099999999999 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:39.821708+0000", 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012571500000000001 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.333 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWrite(3.es1 96/76 ECSubWrite(tid=159, reqid=osd.2.0:159, at_version=96'126, trim_to=0'0, roll_forward_to=96'124))", 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:40.308655+0000", 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.923338013, 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062686333999999996, 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308655+0000", 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308655+0000", 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308656+0000", 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.75e-07 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.308678+0000", 2024-09-15T02:49:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2728e-05 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369529+0000", 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.060850298999999997 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369546+0000", 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7263000000000001e-05 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369648+0000", 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102416 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.369665+0000", 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7062e-05 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.371306+0000", 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016409829999999999 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:40.371341+0000", 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4907999999999998e-05 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.338 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:52.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:52.340 INFO:tasks.ceph.osd.15.smithi191.stderr:2024-09-15T02:49:52.336+0000 7f417e8bc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 22454) UID: 0 2024-09-15T02:49:52.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:49:52.388 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.6. 2024-09-15T02:49:52.399 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-15T02:49:52.440 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:52.440+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:52.451 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-15T02:49:52.452 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:52.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:52.541 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-15T02:49:52.540+0000 7f5a3b058700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22449) UID: 0 2024-09-15T02:49:52.557 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:52.558 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-15T02:49:52.558 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-15T02:49:52.558 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-15T02:49:52.558 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:52.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:52.642 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:52.639+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:52.669 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-15T02:49:52.669 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-15T02:49:52.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-15T02:49:52.669 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-15T02:49:52.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/33 ECSubWriteReply(tid=19, last_complete=38'14, committed=1, applied=1))", 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 70.125846781999996, 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74030938599999996, 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539610+0000", 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539611+0000", 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3680000000000001e-06 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:42.539619+0000", 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278674+0000", 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73905563200000002 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278687+0000", 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.239e-05 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.278784+0000", 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7149000000000001e-05 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.279919+0000", 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001135645 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 39/33 ECSubWriteReply(tid=24, last_complete=39'18, committed=1, applied=1))", 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.078479493000003, 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30338358999999998, 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586977+0000", 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586978+0000", 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.51e-07 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.586981+0000", 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3069999999999998e-06 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890125+0000", 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30314395700000002 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890138+0000", 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2627e-05 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890297+0000", 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015958899999999999 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:43.890361+0000", 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3658999999999995e-05 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:414 3.3s0 3:cebed992:::smithi19125844-27:head [copy-from ver 4] snapc f=[f,e,b,a] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.409593059000002, 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034868557000000001, 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.677 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 414 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255864+0000", 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255866+0000", 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.621e-06 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255879+0000", 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2738999999999999e-05 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255912+0000", 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3099000000000003e-05 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.255927+0000", 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5013e-05 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256006+0000", 2024-09-15T02:49:52.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8688000000000003e-05 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.256016+0000", 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8379999999999997e-06 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262150+0000", 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061347040000000004 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.262292+0000", 2024-09-15T02:49:52.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141595 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.275991+0000", 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013699266999999999 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.282717+0000", 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067257910000000001 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290669+0000", 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079519480000000003 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:48:54.290732+0000", 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3254000000000001e-05 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=77, last_complete=68'67, committed=1, applied=1))", 2024-09-15T02:49:52.683 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.253096018000001, 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.106159008, 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412361+0000", 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412362+0000", 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9999999999999996e-07 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.412370+0000", 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4120000000000001e-06 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517947+0000", 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10557704800000001 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.517957+0000", 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0173999999999999e-05 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518475+0000", 2024-09-15T02:49:52.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051777099999999999 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518520+0000", 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4703000000000002e-05 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:636 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~606208] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.168440871000001, 2024-09-15T02:49:52.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026984608, 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 636 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497016+0000", 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497019+0000", 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.066e-06 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497546+0000", 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052673800000000001 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497580+0000", 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4647999999999998e-05 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497592+0000", 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1669e-05 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497695+0000", 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103031 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.497915+0000", 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000219607 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.499984+0000", 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020692800000000002 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518517+0000", 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018533353999999998 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523868+0000", 2024-09-15T02:49:52.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053511779999999998 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524001+0000", 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132037 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:638 3.3s0 3:c4165a1c:::smithi19125844-22:head [append 0~475136] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:52.693 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.157341271999996, 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.170696291, 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 638 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508116+0000", 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508118+0000", 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6110000000000001e-06 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508908+0000", 2024-09-15T02:49:52.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00078945100000000004 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508947+0000", 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9298999999999998e-05 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.508971+0000", 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4329e-05 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518663+0000", 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0096915070000000002 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.518760+0000", 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7740999999999998e-05 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.519373+0000", 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061209299999999997 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523608+0000", 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042356319999999996 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678746+0000", 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15513796099999999 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678812+0000", 2024-09-15T02:49:52.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5667000000000007e-05 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:639 3.3s0 3:c4165a1c:::smithi19125844-22:head [setxattr _header (57) in=7b] snapc 1d=[1d,18,11,b,a] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.156456749, 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.189219685, 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.699 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 639 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509000+0000", 2024-09-15T02:49:52.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509002+0000", 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7579999999999999e-06 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509010+0000", 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3110000000000006e-06 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509031+0000", 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1234e-05 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509052+0000", 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0962e-05 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522387+0000", 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013334693 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.522514+0000", 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126723 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.523057+0000", 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054339999999999998 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524433+0000", 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001375739 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698072+0000", 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17363879099999999 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698220+0000", 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014807399999999999 2024-09-15T02:49:52.704 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:640 3.3s0 3:c4165a1c:::smithi19125844-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.156369517999998, 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19189399200000001, 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.705 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 640 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509087+0000", 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509090+0000", 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2800000000000002e-06 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509095+0000", 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9679999999999997e-06 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509113+0000", 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8031000000000001e-05 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.509122+0000", 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6509999999999996e-06 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524120+0000", 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014997610999999999 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.524399+0000", 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027887000000000002 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.698989+0000", 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.174589943 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.699138+0000", 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000149333 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.700981+0000", 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018433049999999999 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 68/33 ECSubWriteReply(tid=80, last_complete=68'71, committed=1, applied=1))", 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.138242990000002, 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15161024100000001, 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.710 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527214+0000", 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527215+0000", 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.088e-06 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.527221+0000", 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9909999999999999e-06 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678531+0000", 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15131061000000001 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678545+0000", 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3597e-05 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678666+0000", 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012109599999999999 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:12.678824+0000", 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015785899999999999 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:699 3.12s0 3:4a71560d:::smithi19125844-16:head [rollback 24] snapc 26=[26,25,24,21,1d,18,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:52.714 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.196718179000001, 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.064459370000000002, 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 699 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:52.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468739+0000", 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468741+0000", 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0420000000000001e-06 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468749+0000", 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000002e-06 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468764+0000", 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5353999999999999e-05 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468773+0000", 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7649999999999999e-06 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495474+0000", 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026701740000000002 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.495855+0000", 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038018500000000001 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512214+0000", 2024-09-15T02:49:52.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016359598 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.512474+0000", 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025961499999999999 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.533198+0000", 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020724131 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:703 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.196676367000002, 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056538106999999997, 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 703 2024-09-15T02:49:52.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468780+0000", 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468781+0000", 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4099999999999998e-07 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468786+0000", 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4909999999999999e-06 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468794+0000", 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0229999999999992e-06 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468803+0000", 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1910000000000006e-06 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.500871+0000", 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032068169000000001 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502024+0000", 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001152674 2024-09-15T02:49:52.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525089+0000", 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023065130999999999 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525319+0000", 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022968700000000001 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:704 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.196642431999997, 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056656818999999997, 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.725 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 704 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468814+0000", 2024-09-15T02:49:52.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468815+0000", 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4199999999999995e-07 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468819+0000", 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7359999999999999e-06 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468826+0000", 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0190000000000001e-06 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468829+0000", 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5499999999999999e-06 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502105+0000", 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033275546000000003 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502234+0000", 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012945 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525355+0000", 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023120971000000001 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525471+0000", 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011580499999999999 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:705 3.15s0 3:ae24c343:::smithi19125844-3:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.196622755, 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.056845816, 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.730 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 705 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468834+0000", 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3500000000000002e-07 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468838+0000", 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0690000000000002e-06 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468843+0000", 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7219999999999996e-06 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.468849+0000", 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0340000000000002e-06 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502534+0000", 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033684990999999997 2024-09-15T02:49:52.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.502672+0000", 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137701 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525531+0000", 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022859147 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:18.525680+0000", 2024-09-15T02:49:52.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014871700000000001 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 83/71 ECSubWriteReply(tid=154, last_complete=83'94, committed=1, applied=1))", 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.075921449999999, 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54901637599999997, 2024-09-15T02:49:52.735 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589535+0000", 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589536+0000", 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6800000000000002e-07 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:27.589538+0000", 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.497e-06 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138412+0000", 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54887397000000004 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138426+0000", 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3621999999999999e-05 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138495+0000", 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9417000000000003e-05 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:28.138552+0000", 2024-09-15T02:49:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6601999999999998e-05 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:901 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~638976] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.277952+0000", 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.387504888999999, 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.036249844000000003, 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 901 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.277952+0000", 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.277952+0000", 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.277955+0000", 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4079999999999998e-06 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278491+0000", 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053570099999999997 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278520+0000", 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8688e-05 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278530+0000", 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8519999999999999e-06 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278683+0000", 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015307400000000001 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.278877+0000", 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019389599999999999 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.280761+0000", 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001884658 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.300773+0000", 2024-09-15T02:49:52.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020011856000000001 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314122+0000", 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013348956 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314202+0000", 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9754999999999998e-05 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:902 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~671744] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.281962+0000", 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.383494812999999, 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034372422, 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 902 2024-09-15T02:49:52.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281962+0000", 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281962+0000", 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.281964+0000", 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1090000000000001e-06 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.282424+0000", 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046013199999999998 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.282447+0000", 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2671999999999999e-05 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.282455+0000", 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9540000000000004e-06 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.291848+0000", 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0093927769999999997 2024-09-15T02:49:52.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.291993+0000", 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014574600000000001 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.293159+0000", 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001165346 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.312623+0000", 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019464250999999998 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316207+0000", 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035840730000000001 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316334+0000", 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127362 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.750 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:903 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~802816] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287154+0000", 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.378303158000001, 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.029421439000000001, 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 903 2024-09-15T02:49:52.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287154+0000", 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287154+0000", 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287155+0000", 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.773e-06 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287738+0000", 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00058255399999999999 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287759+0000", 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0987000000000001e-05 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287770+0000", 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1035e-05 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.301023+0000", 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013252788 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.301121+0000", 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.87e-05 2024-09-15T02:49:52.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.302043+0000", 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00092160199999999999 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314264+0000", 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012220846 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316454+0000", 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021897890000000001 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316575+0000", 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121365 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.756 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:904 3.15s0 3:a906e5df:::smithi19125844-36:head [append 0~81920] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287779+0000", 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.377677486, 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.031042132, 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 904 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287779+0000", 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287779+0000", 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287780+0000", 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287817+0000", 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6891999999999998e-05 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287831+0000", 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3550000000000001e-05 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287835+0000", 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-06 2024-09-15T02:49:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.312677+0000", 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024841196999999999 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.312803+0000", 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012656899999999999 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.313234+0000", 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000431261 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.315040+0000", 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018058589999999999 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.318776+0000", 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037358999999999999 2024-09-15T02:49:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.318821+0000", 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5259999999999997e-05 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:905 3.15s0 3:a906e5df:::smithi19125844-36:head [setxattr _header (57) in=7b] snapc 34=[34,33,32,30,2f,2e,2d,2b,28,26,25,24,21,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.377614925, 2024-09-15T02:49:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.031686819999999997, 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 905 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287842+0000", 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3000000000000001e-07 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287846+0000", 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3859999999999999e-06 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287851+0000", 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5439999999999998e-06 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287855+0000", 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.264e-06 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314422+0000", 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026567048999999999 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314491+0000", 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9197999999999997e-05 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started", 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.314867+0000", 2024-09-15T02:49:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037581300000000002 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_committed", 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.316898+0000", 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020310419999999998 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319422+0000", 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002524199 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.319529+0000", 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106795 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4526.0:906 3.15s0 3:a906e5df:::smithi19125844-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-15T02:49:32.287858+0000", 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.377598637999998, 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.033687184000000002, 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-15T02:49:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4526", 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/970165684", 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 906 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287858+0000", 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-15T02:49:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287858+0000", 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287859+0000", 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7099999999999997e-07 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287860+0000", 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.968e-06 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287866+0000", 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9000000000000003e-06 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.287870+0000", 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4479999999999999e-06 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.315082+0000", 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.027211803999999999 2024-09-15T02:49:52.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for rw locks", 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.315179+0000", 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7507e-05 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.320028+0000", 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048489040000000002 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-15T02:49:52.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.320161+0000", 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000132829 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-15T02:49:32.321545+0000", 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013844529999999999 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-15T02:49:52.774 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-15T02:49:52.776 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:52.776 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-15T02:49:52.771+0000 7f7232703700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22122) UID: 0 2024-09-15T02:49:52.877 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-15T02:49:52.875+0000 7f0e3a70d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 22075) UID: 0 2024-09-15T02:49:52.884 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:52.884 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:52.884 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:52.884 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:52.894 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:52.977 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:52.975+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:52.995 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:52.995 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:52.995 INFO:teuthology.orchestra.run.smithi152.stdout: "complaint_time": 30, 2024-09-15T02:49:52.995 INFO:teuthology.orchestra.run.smithi152.stdout: "num_blocked_ops": 0 2024-09-15T02:49:52.995 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:53.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:53.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:49:53.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T02:49:53.078 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-15T02:49:53.075+0000 7fed7f375700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22069) UID: 0 2024-09-15T02:49:53.106 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:53.106 INFO:teuthology.orchestra.run.smithi152.stdout: "size": 20, 2024-09-15T02:49:53.106 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 600, 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:871775f5:::smithi19125844-15:c, version: 48'19, data_included: [0~61440], data_size: 61440, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:871775f5:::smithi19125844-15:c@48'19, size: 245760, copy_subset: [], clone_subset: {}, snapset: c=[]:{c=[b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:245760, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 54.661103546, 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80849678800000002, 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.107 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442895+0000", 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442896+0000", 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0449999999999999e-06 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:58.442986+0000", 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.9937000000000003e-05 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.246998+0000", 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.80401223499999996 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247018+0000", 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9735999999999999e-05 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.247164+0000", 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000146093 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.251392+0000", 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0042277419999999996 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 54/53 [PushOp(3:801a1da7:::smithi19125844-39:e, version: 49'21, data_included: [0~159744], data_size: 159744, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:e@49'21, size: 638976, copy_subset: [], clone_subset: {}, snapset: e=[]:{e=[e,b,a]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:638976, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 53.838373492000002, 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010047222999999999, 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265625+0000", 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265626+0000", 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.325e-06 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265815+0000", 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00018882999999999999 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265866+0000", 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.0565000000000001e-05 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.265876+0000", 2024-09-15T02:49:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0273e-05 2024-09-15T02:49:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.266085+0000", 2024-09-15T02:49:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000209049 2024-09-15T02:49:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.275672+0000", 2024-09-15T02:49:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0095871810000000002 2024-09-15T02:49:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 53.819775528000001, 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0087482150000000002, 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284223+0000", 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1300000000000001e-07 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284497+0000", 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027328299999999998 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284538+0000", 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.1449999999999998e-05 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284547+0000", 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1020000000000003e-06 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.284639+0000", 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.1615999999999998e-05 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:48:59.292971+0000", 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083324509999999994 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:801a1da7:::smithi19125844-39:head, version: 49'25, data_included: [262144~221184], data_size: 221184, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:801a1da7:::smithi19125844-39:head@49'25, size: 1933312, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1933312, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 52.815460156, 2024-09-15T02:49:53.120 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014791191, 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.121 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288538+0000", 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288540+0000", 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9369999999999998e-06 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.122 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.288962+0000", 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042141100000000001 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289058+0000", 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6339e-05 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.123 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289089+0000", 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0867000000000001e-05 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.289255+0000", 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00016596199999999999 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.124 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.125 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.125 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.303330+0000", 2024-09-15T02:49:53.125 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.014074675 2024-09-15T02:49:53.125 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.125 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.125 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.126 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.126 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.126 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.126 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:53.126 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 52.778896502000002, 2024-09-15T02:49:53.126 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012696989000000001, 2024-09-15T02:49:53.127 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.127 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.127 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.127 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.127 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325102+0000", 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.128 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325103+0000", 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.333e-06 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325526+0000", 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000422788 2024-09-15T02:49:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.130 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.130 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.130 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325605+0000", 2024-09-15T02:49:53.130 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8932999999999998e-05 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325624+0000", 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8525999999999998e-05 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.131 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.132 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.132 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.325780+0000", 2024-09-15T02:49:53.132 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00015595999999999999 2024-09-15T02:49:53.132 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.132 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.132 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.133 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.337799+0000", 2024-09-15T02:49:53.133 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012019449 2024-09-15T02:49:53.133 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.133 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.133 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.133 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.134 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.134 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:f, version: 50'31, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:f@50'31, size: 2424832, copy_subset: [], clone_subset: {}, snapset: f=[]:{e=[e,b,a],f=[f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.134 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:53.134 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 52.761531411999997, 2024-09-15T02:49:53.134 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01280455, 2024-09-15T02:49:53.134 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.135 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.135 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.135 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.135 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.135 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:53.135 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.136 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.136 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.136 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.136 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342467+0000", 2024-09-15T02:49:53.136 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.136 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.137 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.137 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.137 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342469+0000", 2024-09-15T02:49:53.137 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6929999999999999e-06 2024-09-15T02:49:53.137 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.137 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.138 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.138 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342850+0000", 2024-09-15T02:49:53.138 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000381462 2024-09-15T02:49:53.138 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.138 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.138 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.139 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342951+0000", 2024-09-15T02:49:53.139 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000100559 2024-09-15T02:49:53.139 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.139 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.139 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.139 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.342971+0000", 2024-09-15T02:49:53.140 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9720000000000001e-05 2024-09-15T02:49:53.140 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.140 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.140 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.140 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.343066+0000", 2024-09-15T02:49:53.140 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5088999999999997e-05 2024-09-15T02:49:53.141 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.141 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.141 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.141 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.355272+0000", 2024-09-15T02:49:53.141 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012206027 2024-09-15T02:49:53.141 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.142 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.142 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.142 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.142 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.142 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.143 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:53.143 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 52.733837321999999, 2024-09-15T02:49:53.143 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.012408221000000001, 2024-09-15T02:49:53.143 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.143 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.143 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.144 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.144 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.144 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:53.144 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.144 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.144 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.145 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.145 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370161+0000", 2024-09-15T02:49:53.145 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.145 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.145 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.145 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.146 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370163+0000", 2024-09-15T02:49:53.146 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6470000000000001e-06 2024-09-15T02:49:53.146 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.146 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.146 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.146 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370553+0000", 2024-09-15T02:49:53.147 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00038988100000000002 2024-09-15T02:49:53.147 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.147 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.147 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.147 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370636+0000", 2024-09-15T02:49:53.147 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2825999999999996e-05 2024-09-15T02:49:53.148 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.148 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.148 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.148 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370655+0000", 2024-09-15T02:49:53.148 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.916e-05 2024-09-15T02:49:53.148 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.149 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.149 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.149 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.370778+0000", 2024-09-15T02:49:53.149 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012348700000000001 2024-09-15T02:49:53.149 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.149 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.150 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.150 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.382570+0000", 2024-09-15T02:49:53.150 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.01179122 2024-09-15T02:49:53.150 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.150 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.150 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.151 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.151 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.151 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDPGPush(3.1s4 55/53 [PushOp(3:8610d5f9:::smithi19125844-1:head, version: 50'35, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8610d5f9:::smithi19125844-1:head@50'35, size: 3817472, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T02:49:53.151 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:53.151 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 52.706745267999999, 2024-09-15T02:49:53.151 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0076400929999999997, 2024-09-15T02:49:53.152 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.152 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.152 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.152 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.152 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.153 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:53.153 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.153 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.153 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.153 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.153 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397253+0000", 2024-09-15T02:49:53.154 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.154 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.154 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.154 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.154 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397255+0000", 2024-09-15T02:49:53.154 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.576e-06 2024-09-15T02:49:53.155 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.155 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.155 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.155 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397632+0000", 2024-09-15T02:49:53.156 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000377265 2024-09-15T02:49:53.156 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.156 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.156 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.156 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397715+0000", 2024-09-15T02:49:53.157 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.3213000000000004e-05 2024-09-15T02:49:53.157 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.157 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.157 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.157 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397743+0000", 2024-09-15T02:49:53.157 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7229999999999998e-05 2024-09-15T02:49:53.158 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.158 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.158 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.158 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.397885+0000", 2024-09-15T02:49:53.158 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014198699999999999 2024-09-15T02:49:53.158 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.159 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.159 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.159 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:00.404893+0000", 2024-09-15T02:49:53.159 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070088219999999996 2024-09-15T02:49:53.159 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.159 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.160 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.160 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.160 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.160 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 60/53 ECSubWrite(tid=67, reqid=client.4526.0:522, at_version=60'31, trim_to=0'0, roll_forward_to=59'22))", 2024-09-15T02:49:53.160 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:53.161 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 48.836450904000003, 2024-09-15T02:49:53.161 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43332960599999998, 2024-09-15T02:49:53.161 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.161 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.161 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.161 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.162 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.162 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:53.162 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.162 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.162 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.163 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267548+0000", 2024-09-15T02:49:53.164 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.8200000000000001e-07 2024-09-15T02:49:53.164 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.164 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.164 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.164 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.267569+0000", 2024-09-15T02:49:53.164 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0713e-05 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699630+0000", 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.43206165099999999 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699644+0000", 2024-09-15T02:49:53.165 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3274e-05 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699794+0000", 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000150719 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.699816+0000", 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1895999999999998e-05 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700821+0000", 2024-09-15T02:49:53.166 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0010047210000000001 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:04.700877+0000", 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6249999999999998e-05 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 72/71 ECSubWrite(tid=403, reqid=osd.13.0:403, at_version=72'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 36.845322465999999, 2024-09-15T02:49:53.167 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0085463759999999996, 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258676+0000", 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8500000000000002e-07 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258694+0000", 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7482e-05 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258812+0000", 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000118245 2024-09-15T02:49:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.258817+0000", 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4379999999999997e-06 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259246+0000", 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00042895599999999998 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.259256+0000", 2024-09-15T02:49:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0818e-05 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.267192+0000", 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0079355799999999994 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:16.267223+0000", 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0571999999999998e-05 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.171 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs3 78/76 ECSubWrite(tid=131, reqid=client.4526.0:756, at_version=78'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 30.606830801000001, 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.021673943000000001, 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.172 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.497168+0000", 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8700000000000002e-07 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499075+0000", 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.001907176 2024-09-15T02:49:53.173 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499139+0000", 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3250000000000006e-05 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499151+0000", 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2371000000000001e-05 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.174 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499260+0000", 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000109064 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.499277+0000", 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.6659000000000001e-05 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.518812+0000", 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.019535858 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.175 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:22.518842+0000", 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9278e-05 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs3 93/76 ECSubWrite(tid=156, reqid=client.4526.0:952, at_version=93'25, trim_to=0'0, roll_forward_to=85'23))", 2024-09-15T02:49:53.176 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:37.285699+0000", 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 15.818299787000001, 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.041223722999999997, 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.285699+0000", 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.177 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.285699+0000", 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.285699+0000", 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.6600000000000002e-07 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.289493+0000", 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0037933580000000001 2024-09-15T02:49:53.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.289555+0000", 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2638000000000006e-05 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.289568+0000", 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2357e-05 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.179 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.289712+0000", 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00014410599999999999 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.289740+0000", 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.8396000000000001e-05 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.326877+0000", 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.037136470999999997 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.180 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:37.326923+0000", 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.6031e-05 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 95/73 ECSubWriteReply(tid=6, last_complete=95'69, committed=1, applied=1))", 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:39.246043+0000", 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 13.857955196000001, 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.67781176300000001, 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.181 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.246043+0000", 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.246043+0000", 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.246045+0000", 2024-09-15T02:49:53.182 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5859999999999999e-06 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.246055+0000", 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.7399999999999999e-06 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.922996+0000", 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.67694131700000004 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.183 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.923016+0000", 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9896999999999999e-05 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.923116+0000", 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.9842999999999999e-05 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.923855+0000", 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00073937999999999999 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 95/71 ECSubWrite(tid=539, reqid=client.4526.0:958, at_version=95'82, trim_to=0'0, roll_forward_to=85'80))", 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:39.270568+0000", 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 13.833430781000001, 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0077078670000000002, 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.270568+0000", 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.185 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.270568+0000", 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.270568+0000", 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.48e-07 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.271099+0000", 2024-09-15T02:49:53.186 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00053149100000000004 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.271178+0000", 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.8444999999999999e-05 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.271187+0000", 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.329e-06 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.187 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.271240+0000", 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2571000000000002e-05 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.271247+0000", 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.0419999999999997e-06 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.278249+0000", 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0070021950000000001 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:39.278276+0000", 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.6645999999999999e-05 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 96/71 ECSubWrite(tid=551, reqid=osd.13.0:551, at_version=96'88, trim_to=0'0, roll_forward_to=95'86))", 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:40.286774+0000", 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.817225083, 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011613137000000001, 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.189 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.286774+0000", 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.286774+0000", 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.286774+0000", 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.5000000000000003e-07 2024-09-15T02:49:53.190 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.286857+0000", 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2441000000000006e-05 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.286982+0000", 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00012575999999999999 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.287001+0000", 2024-09-15T02:49:53.191 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.8377e-05 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.287102+0000", 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.000101829 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.287118+0000", 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.5727e-05 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.192 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298357+0000", 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.011238428999999999 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298387+0000", 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.0023999999999999e-05 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.193 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.1s4 96/71 ECSubWrite(tid=553, reqid=osd.13.0:553, at_version=96'90, trim_to=0'0, roll_forward_to=95'86))", 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:40.287983+0000", 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.816015846999999, 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.010424958, 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.287983+0000", 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.194 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.287983+0000", 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.287983+0000", 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.6300000000000005e-07 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.288003+0000", 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9519999999999999e-05 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.195 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.288073+0000", 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.9683000000000004e-05 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.288077+0000", 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9540000000000003e-06 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.288691+0000", 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00061378400000000003 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.288702+0000", 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0563e-05 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298394+0000", 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0096922840000000007 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298408+0000", 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3607e-05 2024-09-15T02:49:53.197 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 96/71 ECSubWrite(tid=156, reqid=osd.14.0:156, at_version=96'59, trim_to=0'0, roll_forward_to=95'57))", 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:40.289057+0000", 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.814941882999999, 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.009371898, 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.198 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289057+0000", 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289057+0000", 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289057+0000", 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.35e-07 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.199 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289084+0000", 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7158e-05 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289143+0000", 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.8795000000000002e-05 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289150+0000", 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.72e-06 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.200 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289873+0000", 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00072340499999999999 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.289882+0000", 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2960000000000004e-06 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298411+0000", 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0085286449999999996 2024-09-15T02:49:53.201 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298429+0000", 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.7643999999999999e-05 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.cs3 96/76 ECSubWrite(tid=168, reqid=osd.0.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=96'31))", 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:40.298056+0000", 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.805942612999999, 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0083328389999999999, 2024-09-15T02:49:53.202 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298056+0000", 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298056+0000", 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298057+0000", 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.2099999999999995e-07 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298070+0000", 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3192e-05 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298105+0000", 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5277000000000001e-05 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.204 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298112+0000", 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.8539999999999999e-06 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298161+0000", 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9008000000000002e-05 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.298170+0000", 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.8969999999999997e-06 2024-09-15T02:49:53.205 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.306356+0000", 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00818581 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.306389+0000", 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2879999999999997e-05 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.206 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.fs5 96/53 ECSubWrite(tid=169, reqid=osd.0.0:166, at_version=0'0, trim_to=0'0, roll_forward_to=96'86))", 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:40.299083+0000", 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.804916050999999, 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0074166620000000001, 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299083+0000", 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.207 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299083+0000", 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299083+0000", 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.9900000000000001e-07 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299093+0000", 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0052e-05 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.208 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299123+0000", 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.9759000000000001e-05 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299129+0000", 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5019999999999998e-06 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299174+0000", 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 4.4367999999999999e-05 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.209 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299180+0000", 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.5690000000000001e-06 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.306435+0000", 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0072543420000000004 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.306499+0000", 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.4571000000000002e-05 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.210 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(3.9s6 96/71 ECSubWrite(tid=157, reqid=osd.14.0:156, at_version=0'0, trim_to=0'0, roll_forward_to=96'59))", 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-15T02:49:40.299255+0000", 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 12.80474332, 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0072131039999999997, 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299255+0000", 2024-09-15T02:49:53.211 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299255+0000", 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299256+0000", 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6599999999999995e-07 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.212 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299288+0000", 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2129999999999999e-05 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299341+0000", 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.2976000000000002e-05 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299349+0000", 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.5619999999999998e-06 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299408+0000", 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9768000000000001e-05 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.299416+0000", 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.2370000000000003e-06 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.214 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_committed", 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.306437+0000", 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.007020854 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "done", 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-15T02:49:40.306468+0000", 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.1810999999999997e-05 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-15T02:49:53.215 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:53.216 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-15T02:49:53.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:53.218 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:53.214+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:53.228 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-15T02:49:53.254 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-15T02:49:53.284 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-15T02:49:53.305 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-15T02:49:53.317 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:53.314+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:53.325 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-15T02:49:53.325 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-15T02:49:53.325 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-15T02:49:53.325 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-15T02:49:53.334 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:49:53.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-15T02:49:53.358 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:49:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:49:53.372 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-15T02:49:52.977446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133415475,"num_objects":125,"num_object_clones":75,"num_object_copies":855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":12,"num_read":355,"num_read_kb":151085,"num_write":675,"num_write_kb":95802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":40222720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":4749968,"kb_used_data":251652,"kb_used_omap":25,"kb_used_meta":4498086,"kb_avail":1410827632,"statfs":{"total":1449551462400,"available":1444687495168,"internally_reserved":0,"allocated":257691648,"data_stored":349625980,"data_compressed":2457414,"data_compressed_allocated":5132288,"data_compressed_original":11157504,"omap_allocated":25640,"internal_metadata":4606041048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.640508"},"pg_stats":[{"pgid":"3.19","version":"97'73","reported_seq":162,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.289410+0000","last_change":"2024-09-15T02:49:40.305146+0000","last_active":"2024-09-15T02:49:49.289410+0000","last_peered":"2024-09-15T02:49:49.289410+0000","last_clean":"2024-09-15T02:49:49.289410+0000","last_became_active":"2024-09-15T02:49:18.470335+0000","last_became_peered":"2024-09-15T02:49:18.470335+0000","last_unstale":"2024-09-15T02:49:49.289410+0000","last_undegraded":"2024-09-15T02:49:49.289410+0000","last_fullsized":"2024-09-15T02:49:49.289410+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'51","last_scrub_stamp":"2024-09-15T02:49:21.298540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:21.298540+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:25.010610+0000","scrub_duration":0.0083571319999999998,"objects_trimmed":0,"snaptrim_duration":0.0094649729999999998,"stat_sum":{"num_bytes":7979008,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":4496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,0,3,14,8,11],"acting":[9,1,0,3,14,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"97'70","reported_seq":160,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.291143+0000","last_change":"2024-09-15T02:49:40.314488+0000","last_active":"2024-09-15T02:49:49.291143+0000","last_peered":"2024-09-15T02:49:49.291143+0000","last_clean":"2024-09-15T02:49:49.291143+0000","last_became_active":"2024-09-15T02:49:21.472717+0000","last_became_peered":"2024-09-15T02:49:21.472717+0000","last_unstale":"2024-09-15T02:49:49.291143+0000","last_undegraded":"2024-09-15T02:49:49.291143+0000","last_fullsized":"2024-09-15T02:49:49.291143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'56","last_scrub_stamp":"2024-09-15T02:49:29.290881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.290881+0000","objects_scrubbed":3,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:37.053128+0000","scrub_duration":0.0059159260000000002,"objects_trimmed":0,"snaptrim_duration":0.017248988,"stat_sum":{"num_bytes":4227072,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1,13,11,12,5],"acting":[3,9,1,13,11,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"96'123","reported_seq":228,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.244436+0000","last_change":"2024-09-15T02:49:40.299156+0000","last_active":"2024-09-15T02:49:49.244436+0000","last_peered":"2024-09-15T02:49:49.244436+0000","last_clean":"2024-09-15T02:49:49.244436+0000","last_became_active":"2024-09-15T02:49:22.482618+0000","last_became_peered":"2024-09-15T02:49:22.482618+0000","last_unstale":"2024-09-15T02:49:49.244436+0000","last_undegraded":"2024-09-15T02:49:49.244436+0000","last_fullsized":"2024-09-15T02:49:49.244436+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020290420999999999,"stat_sum":{"num_bytes":3563520,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":46,"num_read_kb":25538,"num_write":66,"num_write_kb":5232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6193152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12,10,7,4,15,3],"acting":[11,12,10,7,4,15,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.16","version":"85'34","reported_seq":101,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.274527+0000","last_change":"2024-09-15T02:49:40.292053+0000","last_active":"2024-09-15T02:49:49.274527+0000","last_peered":"2024-09-15T02:49:49.274527+0000","last_clean":"2024-09-15T02:49:49.274527+0000","last_became_active":"2024-09-15T02:49:18.462615+0000","last_became_peered":"2024-09-15T02:49:18.462615+0000","last_unstale":"2024-09-15T02:49:49.274527+0000","last_undegraded":"2024-09-15T02:49:49.274527+0000","last_fullsized":"2024-09-15T02:49:49.274527+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'34","last_scrub_stamp":"2024-09-15T02:49:30.409672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:30.409672+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:48.545363+0000","scrub_duration":0.0056793069999999998,"objects_trimmed":0,"snaptrim_duration":0.00689239,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,11,9,5,7,15,10],"acting":[12,11,9,5,7,15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.15","version":"96'67","reported_seq":181,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.297359+0000","last_change":"2024-09-15T02:49:40.310474+0000","last_active":"2024-09-15T02:49:49.297359+0000","last_peered":"2024-09-15T02:49:49.297359+0000","last_clean":"2024-09-15T02:49:49.297359+0000","last_became_active":"2024-09-15T02:49:21.482421+0000","last_became_peered":"2024-09-15T02:49:21.482421+0000","last_unstale":"2024-09-15T02:49:49.297359+0000","last_undegraded":"2024-09-15T02:49:49.297359+0000","last_fullsized":"2024-09-15T02:49:49.297359+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'57","last_scrub_stamp":"2024-09-15T02:49:30.515515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:30.515515+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:39.130392+0000","scrub_duration":0.58297202400000003,"objects_trimmed":0,"snaptrim_duration":0.032648980000000001,"stat_sum":{"num_bytes":6864896,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5,8,7,4,2,3],"acting":[15,5,8,7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.14","version":"96'4","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T02:49:44.242175+0000","last_change":"2024-09-15T02:49:40.295151+0000","last_active":"2024-09-15T02:49:44.242175+0000","last_peered":"2024-09-15T02:49:44.242175+0000","last_clean":"2024-09-15T02:49:44.242175+0000","last_became_active":"2024-09-15T02:49:18.463893+0000","last_became_peered":"2024-09-15T02:49:18.463893+0000","last_unstale":"2024-09-15T02:49:44.242175+0000","last_undegraded":"2024-09-15T02:49:44.242175+0000","last_fullsized":"2024-09-15T02:49:44.242175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'2","last_scrub_stamp":"2024-09-15T02:49:32.931133+0000","last_deep_scrub":"36'2","last_deep_scrub_stamp":"2024-09-15T02:49:32.931133+0000","last_clean_scrub_stamp":"2024-09-15T02:49:32.931133+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:59.917241+0000","scrub_duration":0.222807329,"objects_trimmed":0,"snaptrim_duration":0.0070962999999999998,"stat_sum":{"num_bytes":573440,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1741,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,12,15,11,9,13],"acting":[4,7,12,15,11,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-15T02:49:45.765206+0000","last_change":"2024-09-15T02:48:52.652454+0000","last_active":"2024-09-15T02:49:45.765206+0000","last_peered":"2024-09-15T02:49:45.765206+0000","last_clean":"2024-09-15T02:49:45.765206+0000","last_became_active":"2024-09-15T02:48:52.652176+0000","last_became_peered":"2024-09-15T02:48:52.652176+0000","last_unstale":"2024-09-15T02:49:45.765206+0000","last_undegraded":"2024-09-15T02:49:45.765206+0000","last_fullsized":"2024-09-15T02:49:45.765206+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:56.216530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"85'34","reported_seq":262,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.246657+0000","last_change":"2024-09-15T02:49:40.301626+0000","last_active":"2024-09-15T02:49:49.246657+0000","last_peered":"2024-09-15T02:49:49.246657+0000","last_clean":"2024-09-15T02:49:49.246657+0000","last_became_active":"2024-09-15T02:49:16.262927+0000","last_became_peered":"2024-09-15T02:49:16.262927+0000","last_unstale":"2024-09-15T02:49:49.246657+0000","last_undegraded":"2024-09-15T02:49:49.246657+0000","last_fullsized":"2024-09-15T02:49:49.246657+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'32","last_scrub_stamp":"2024-09-15T02:49:17.616353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:17.616353+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:27.488386+0000","scrub_duration":0.0062733370000000004,"objects_trimmed":0,"snaptrim_duration":0.021273767999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":112,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.290088+0000","last_change":"2024-09-15T02:49:43.343438+0000","last_active":"2024-09-15T02:49:49.290088+0000","last_peered":"2024-09-15T02:49:49.290088+0000","last_clean":"2024-09-15T02:49:49.290088+0000","last_became_active":"2024-09-15T02:49:21.466842+0000","last_became_peered":"2024-09-15T02:49:21.466842+0000","last_unstale":"2024-09-15T02:49:49.290088+0000","last_undegraded":"2024-09-15T02:49:49.290088+0000","last_fullsized":"2024-09-15T02:49:49.290088+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:55.005749+0000","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"96'125","reported_seq":392,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.246810+0000","last_change":"2024-09-15T02:49:43.617516+0000","last_active":"2024-09-15T02:49:49.246810+0000","last_peered":"2024-09-15T02:49:49.246810+0000","last_clean":"2024-09-15T02:49:49.246810+0000","last_became_active":"2024-09-15T02:49:24.496338+0000","last_became_peered":"2024-09-15T02:49:24.496338+0000","last_unstale":"2024-09-15T02:49:49.246810+0000","last_undegraded":"2024-09-15T02:49:49.246810+0000","last_fullsized":"2024-09-15T02:49:49.246810+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'125","last_scrub_stamp":"2024-09-15T02:49:43.617378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.617378+0000","objects_scrubbed":5,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:54.906816+0000","scrub_duration":0.0069573259999999998,"objects_trimmed":0,"snaptrim_duration":0.033530000999999997,"stat_sum":{"num_bytes":7307264,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,7,1,11,5,0],"acting":[13,4,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.291578+0000","last_change":"2024-09-15T02:49:42.473084+0000","last_active":"2024-09-15T02:49:49.291578+0000","last_peered":"2024-09-15T02:49:49.291578+0000","last_clean":"2024-09-15T02:49:49.291578+0000","last_became_active":"2024-09-15T02:49:21.465616+0000","last_became_peered":"2024-09-15T02:49:21.465616+0000","last_unstale":"2024-09-15T02:49:49.291578+0000","last_undegraded":"2024-09-15T02:49:49.291578+0000","last_fullsized":"2024-09-15T02:49:49.291578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:11.025273+0000","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,13],"acting":[5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"36'2","reported_seq":204,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.292436+0000","last_change":"2024-09-15T02:49:40.287039+0000","last_active":"2024-09-15T02:49:49.292436+0000","last_peered":"2024-09-15T02:49:49.292436+0000","last_clean":"2024-09-15T02:49:49.292436+0000","last_became_active":"2024-09-15T02:49:16.256552+0000","last_became_peered":"2024-09-15T02:49:16.256552+0000","last_unstale":"2024-09-15T02:49:49.292436+0000","last_undegraded":"2024-09-15T02:49:49.292436+0000","last_fullsized":"2024-09-15T02:49:49.292436+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'2","last_scrub_stamp":"2024-09-15T02:49:17.582201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:17.582201+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:31.619423+0000","scrub_duration":0.0051105170000000002,"objects_trimmed":0,"snaptrim_duration":0.0024865400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":98,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.293011+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:49:49.293011+0000","last_peered":"2024-09-15T02:49:49.293011+0000","last_clean":"2024-09-15T02:49:49.293011+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:49:49.293011+0000","last_undegraded":"2024-09-15T02:49:49.293011+0000","last_fullsized":"2024-09-15T02:49:49.293011+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"88'57","reported_seq":294,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.246503+0000","last_change":"2024-09-15T02:49:40.281489+0000","last_active":"2024-09-15T02:49:49.246503+0000","last_peered":"2024-09-15T02:49:49.246503+0000","last_clean":"2024-09-15T02:49:49.246503+0000","last_became_active":"2024-09-15T02:49:21.482725+0000","last_became_peered":"2024-09-15T02:49:21.482725+0000","last_unstale":"2024-09-15T02:49:49.246503+0000","last_undegraded":"2024-09-15T02:49:49.246503+0000","last_fullsized":"2024-09-15T02:49:49.246503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'57","last_scrub_stamp":"2024-09-15T02:49:34.649254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:34.649254+0000","objects_scrubbed":2,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:37.653161+0000","scrub_duration":0.0069899530000000001,"objects_trimmed":0,"snaptrim_duration":0.001480437,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,15],"acting":[8,13,5,11,9,1,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"96'104","reported_seq":315,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.297183+0000","last_change":"2024-09-15T02:49:40.307711+0000","last_active":"2024-09-15T02:49:49.297183+0000","last_peered":"2024-09-15T02:49:49.297183+0000","last_clean":"2024-09-15T02:49:49.297183+0000","last_became_active":"2024-09-15T02:49:16.251895+0000","last_became_peered":"2024-09-15T02:49:16.251895+0000","last_unstale":"2024-09-15T02:49:49.297183+0000","last_undegraded":"2024-09-15T02:49:49.297183+0000","last_fullsized":"2024-09-15T02:49:49.297183+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'90","last_scrub_stamp":"2024-09-15T02:49:21.457038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:21.457038+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:25.998414+0000","scrub_duration":0.30840548299999998,"objects_trimmed":0,"snaptrim_duration":0.030237591000000001,"stat_sum":{"num_bytes":6324224,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":18739,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":83,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.290606+0000","last_change":"2024-09-15T02:48:52.654051+0000","last_active":"2024-09-15T02:49:49.290606+0000","last_peered":"2024-09-15T02:49:49.290606+0000","last_clean":"2024-09-15T02:49:49.290606+0000","last_became_active":"2024-09-15T02:48:52.653285+0000","last_became_peered":"2024-09-15T02:48:52.653285+0000","last_unstale":"2024-09-15T02:49:49.290606+0000","last_undegraded":"2024-09-15T02:49:49.290606+0000","last_fullsized":"2024-09-15T02:49:49.290606+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:48:26.377691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"97'36","reported_seq":251,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.292730+0000","last_change":"2024-09-15T02:49:40.290003+0000","last_active":"2024-09-15T02:49:49.292730+0000","last_peered":"2024-09-15T02:49:49.292730+0000","last_clean":"2024-09-15T02:49:49.292730+0000","last_became_active":"2024-09-15T02:49:16.260290+0000","last_became_peered":"2024-09-15T02:49:16.260290+0000","last_unstale":"2024-09-15T02:49:49.292730+0000","last_undegraded":"2024-09-15T02:49:49.292730+0000","last_fullsized":"2024-09-15T02:49:49.292730+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'34","last_scrub_stamp":"2024-09-15T02:49:31.403074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.403074+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:48.299772+0000","scrub_duration":0.99022798000000001,"objects_trimmed":0,"snaptrim_duration":0.001687923,"stat_sum":{"num_bytes":5505024,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.294258+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:49:49.294258+0000","last_peered":"2024-09-15T02:49:49.294258+0000","last_clean":"2024-09-15T02:49:49.294258+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:49:49.294258+0000","last_undegraded":"2024-09-15T02:49:49.294258+0000","last_fullsized":"2024-09-15T02:49:49.294258+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"97'93","reported_seq":349,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.246136+0000","last_change":"2024-09-15T02:49:40.300811+0000","last_active":"2024-09-15T02:49:49.246136+0000","last_peered":"2024-09-15T02:49:49.246136+0000","last_clean":"2024-09-15T02:49:49.246136+0000","last_became_active":"2024-09-15T02:49:16.255255+0000","last_became_peered":"2024-09-15T02:49:16.255255+0000","last_unstale":"2024-09-15T02:49:49.246136+0000","last_undegraded":"2024-09-15T02:49:49.246136+0000","last_fullsized":"2024-09-15T02:49:49.246136+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'80","last_scrub_stamp":"2024-09-15T02:49:33.693549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:33.693549+0000","objects_scrubbed":5,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:56.782722+0000","scrub_duration":0.011892081000000001,"objects_trimmed":0,"snaptrim_duration":0.020733623999999999,"stat_sum":{"num_bytes":10240000,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":4608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.291221+0000","last_change":"2024-09-15T02:49:44.853425+0000","last_active":"2024-09-15T02:49:49.291221+0000","last_peered":"2024-09-15T02:49:49.291221+0000","last_clean":"2024-09-15T02:49:49.291221+0000","last_became_active":"2024-09-15T02:49:21.465721+0000","last_became_peered":"2024-09-15T02:49:21.465721+0000","last_unstale":"2024-09-15T02:49:49.291221+0000","last_undegraded":"2024-09-15T02:49:49.291221+0000","last_fullsized":"2024-09-15T02:49:49.291221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.853218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.853218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.097187+0000","scrub_duration":0.65334804099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"70'37","reported_seq":236,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.244349+0000","last_change":"2024-09-15T02:49:45.039442+0000","last_active":"2024-09-15T02:49:49.244349+0000","last_peered":"2024-09-15T02:49:49.244349+0000","last_clean":"2024-09-15T02:49:49.244349+0000","last_became_active":"2024-09-15T02:49:16.263776+0000","last_became_peered":"2024-09-15T02:49:16.263776+0000","last_unstale":"2024-09-15T02:49:49.244349+0000","last_undegraded":"2024-09-15T02:49:49.244349+0000","last_fullsized":"2024-09-15T02:49:49.244349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'37","last_scrub_stamp":"2024-09-15T02:49:45.039300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:45.039300+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.168919+0000","scrub_duration":0.0042200969999999999,"objects_trimmed":0,"snaptrim_duration":0.003929246,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":102,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.243655+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:49:49.243655+0000","last_peered":"2024-09-15T02:49:49.243655+0000","last_clean":"2024-09-15T02:49:49.243655+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:49:49.243655+0000","last_undegraded":"2024-09-15T02:49:49.243655+0000","last_fullsized":"2024-09-15T02:49:49.243655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":225,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.243502+0000","last_change":"2024-09-15T02:48:24.408991+0000","last_active":"2024-09-15T02:49:49.243502+0000","last_peered":"2024-09-15T02:49:49.243502+0000","last_clean":"2024-09-15T02:49:49.243502+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:49:49.243502+0000","last_undegraded":"2024-09-15T02:49:49.243502+0000","last_fullsized":"2024-09-15T02:49:49.243502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:48:23.322558+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"96'69","reported_seq":308,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.243708+0000","last_change":"2024-09-15T02:49:40.299245+0000","last_active":"2024-09-15T02:49:49.243708+0000","last_peered":"2024-09-15T02:49:49.243708+0000","last_clean":"2024-09-15T02:49:49.243708+0000","last_became_active":"2024-09-15T02:49:21.477251+0000","last_became_peered":"2024-09-15T02:49:21.477251+0000","last_unstale":"2024-09-15T02:49:49.243708+0000","last_undegraded":"2024-09-15T02:49:49.243708+0000","last_fullsized":"2024-09-15T02:49:49.243708+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022552333000000001,"stat_sum":{"num_bytes":7692288,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":18128,"num_write":43,"num_write_kb":7960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,3],"acting":[11,1,2,15,14,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.9","version":"96'59","reported_seq":317,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.292401+0000","last_change":"2024-09-15T02:49:40.299892+0000","last_active":"2024-09-15T02:49:49.292401+0000","last_peered":"2024-09-15T02:49:49.292401+0000","last_clean":"2024-09-15T02:49:49.292401+0000","last_became_active":"2024-09-15T02:49:16.253291+0000","last_became_peered":"2024-09-15T02:49:16.253291+0000","last_unstale":"2024-09-15T02:49:49.292401+0000","last_undegraded":"2024-09-15T02:49:49.292401+0000","last_fullsized":"2024-09-15T02:49:49.292401+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'47","last_scrub_stamp":"2024-09-15T02:49:17.495488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:17.495488+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:43.440276+0000","scrub_duration":0.005330581,"objects_trimmed":0,"snaptrim_duration":0.012017310999999999,"stat_sum":{"num_bytes":442368,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":2594,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.a","version":"97'91","reported_seq":309,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.273906+0000","last_change":"2024-09-15T02:49:40.311600+0000","last_active":"2024-09-15T02:49:49.273906+0000","last_peered":"2024-09-15T02:49:49.273906+0000","last_clean":"2024-09-15T02:49:49.273906+0000","last_became_active":"2024-09-15T02:49:21.475789+0000","last_became_peered":"2024-09-15T02:49:21.475789+0000","last_unstale":"2024-09-15T02:49:49.273906+0000","last_undegraded":"2024-09-15T02:49:49.273906+0000","last_fullsized":"2024-09-15T02:49:49.273906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:59.254232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026301155999999999,"stat_sum":{"num_bytes":9928704,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":32,"num_read_kb":24780,"num_write":34,"num_write_kb":10144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5849088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,0,8],"acting":[12,4,7,3,14,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.b","version":"97'128","reported_seq":356,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.246252+0000","last_change":"2024-09-15T02:49:40.309431+0000","last_active":"2024-09-15T02:49:49.246252+0000","last_peered":"2024-09-15T02:49:49.246252+0000","last_clean":"2024-09-15T02:49:49.246252+0000","last_became_active":"2024-09-15T02:49:21.476878+0000","last_became_peered":"2024-09-15T02:49:21.476878+0000","last_unstale":"2024-09-15T02:49:49.246252+0000","last_undegraded":"2024-09-15T02:49:49.246252+0000","last_fullsized":"2024-09-15T02:49:49.246252+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:05.374384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029734667999999999,"stat_sum":{"num_bytes":17072128,"num_objects":15,"num_object_clones":10,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":43,"num_read_kb":20416,"num_write":92,"num_write_kb":19200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":10584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,14,4,7],"acting":[13,1,11,15,14,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"97'33","reported_seq":256,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.247007+0000","last_change":"2024-09-15T02:49:40.301198+0000","last_active":"2024-09-15T02:49:49.247007+0000","last_peered":"2024-09-15T02:49:49.247007+0000","last_clean":"2024-09-15T02:49:49.247007+0000","last_became_active":"2024-09-15T02:49:21.464619+0000","last_became_peered":"2024-09-15T02:49:21.464619+0000","last_unstale":"2024-09-15T02:49:49.247007+0000","last_undegraded":"2024-09-15T02:49:49.247007+0000","last_fullsized":"2024-09-15T02:49:49.247007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012996804000000001,"stat_sum":{"num_bytes":9535488,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":10,"num_write_kb":9312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3899392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,14,1,9],"acting":[0,13,12,10,14,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"96'27","reported_seq":212,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.274138+0000","last_change":"2024-09-15T02:49:40.311500+0000","last_active":"2024-09-15T02:49:49.274138+0000","last_peered":"2024-09-15T02:49:49.274138+0000","last_clean":"2024-09-15T02:49:49.274138+0000","last_became_active":"2024-09-15T02:48:38.545950+0000","last_became_peered":"2024-09-15T02:48:38.545950+0000","last_unstale":"2024-09-15T02:49:49.274138+0000","last_undegraded":"2024-09-15T02:49:49.274138+0000","last_fullsized":"2024-09-15T02:49:49.274138+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:58.729969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026434485000000001,"stat_sum":{"num_bytes":2883584,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":10518,"num_write":10,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.e","version":"96'126","reported_seq":373,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.291549+0000","last_change":"2024-09-15T02:49:42.333975+0000","last_active":"2024-09-15T02:49:49.291549+0000","last_peered":"2024-09-15T02:49:49.291549+0000","last_clean":"2024-09-15T02:49:49.291549+0000","last_became_active":"2024-09-15T02:49:21.473900+0000","last_became_peered":"2024-09-15T02:49:21.473900+0000","last_unstale":"2024-09-15T02:49:49.291549+0000","last_undegraded":"2024-09-15T02:49:49.291549+0000","last_fullsized":"2024-09-15T02:49:49.291549+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'126","last_scrub_stamp":"2024-09-15T02:49:42.333798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.333798+0000","objects_scrubbed":11,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:44.360778+0000","scrub_duration":0.016797395,"objects_trimmed":0,"snaptrim_duration":0.088800501000000004,"stat_sum":{"num_bytes":12419072,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":38,"num_read_kb":20233,"num_write":127,"num_write_kb":14800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,7,4,13,0,5],"acting":[2,11,7,4,13,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"96'86","reported_seq":304,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.247356+0000","last_change":"2024-09-15T02:49:40.299640+0000","last_active":"2024-09-15T02:49:49.247356+0000","last_peered":"2024-09-15T02:49:49.247356+0000","last_clean":"2024-09-15T02:49:49.247356+0000","last_became_active":"2024-09-15T02:48:58.322126+0000","last_became_peered":"2024-09-15T02:48:58.322126+0000","last_unstale":"2024-09-15T02:49:49.247356+0000","last_undegraded":"2024-09-15T02:49:49.247356+0000","last_fullsized":"2024-09-15T02:49:49.247356+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:48:37.476646+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:03.904159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011083486,"stat_sum":{"num_bytes":8355840,"num_objects":10,"num_object_clones":6,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":8317,"num_write":52,"num_write_kb":8160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"70'30","reported_seq":96,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-15T02:49:45.764584+0000","last_change":"2024-09-15T02:49:43.218363+0000","last_active":"2024-09-15T02:49:45.764584+0000","last_peered":"2024-09-15T02:49:45.764584+0000","last_clean":"2024-09-15T02:49:45.764584+0000","last_became_active":"2024-09-15T02:49:21.475674+0000","last_became_peered":"2024-09-15T02:49:21.475674+0000","last_unstale":"2024-09-15T02:49:45.764584+0000","last_undegraded":"2024-09-15T02:49:45.764584+0000","last_fullsized":"2024-09-15T02:49:45.764584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'30","last_scrub_stamp":"2024-09-15T02:49:43.218104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.218104+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:12.469465+0000","scrub_duration":0.0051695639999999998,"objects_trimmed":0,"snaptrim_duration":0.0020836539999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8,15,5,9,0],"acting":[7,11,8,15,5,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.11","version":"96'71","reported_seq":108,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.291046+0000","last_change":"2024-09-15T02:49:40.308678+0000","last_active":"2024-09-15T02:49:49.291046+0000","last_peered":"2024-09-15T02:49:49.291046+0000","last_clean":"2024-09-15T02:49:49.291046+0000","last_became_active":"2024-09-15T02:49:18.464014+0000","last_became_peered":"2024-09-15T02:49:18.464014+0000","last_unstale":"2024-09-15T02:49:49.291046+0000","last_undegraded":"2024-09-15T02:49:49.291046+0000","last_fullsized":"2024-09-15T02:49:49.291046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'65","last_scrub_stamp":"2024-09-15T02:49:31.195982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.195982+0000","objects_scrubbed":2,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:54.263248+0000","scrub_duration":0.0066166080000000004,"objects_trimmed":0,"snaptrim_duration":0.020407696999999999,"stat_sum":{"num_bytes":1343488,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0,3,5,7,1],"acting":[10,4,0,3,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.12","version":"95'56","reported_seq":185,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.295333+0000","last_change":"2024-09-15T02:49:40.279007+0000","last_active":"2024-09-15T02:49:49.295333+0000","last_peered":"2024-09-15T02:49:49.295333+0000","last_clean":"2024-09-15T02:49:49.295333+0000","last_became_active":"2024-09-15T02:49:22.511352+0000","last_became_peered":"2024-09-15T02:49:22.511352+0000","last_unstale":"2024-09-15T02:49:49.295333+0000","last_undegraded":"2024-09-15T02:49:49.295333+0000","last_fullsized":"2024-09-15T02:49:49.295333+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'54","last_scrub_stamp":"2024-09-15T02:49:31.964027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.964027+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:45.301571+0000","scrub_duration":0.016273553999999999,"objects_trimmed":0,"snaptrim_duration":0.001568769,"stat_sum":{"num_bytes":8028160,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,5,9,12,2,7],"acting":[15,8,5,9,12,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.13","version":"97'94","reported_seq":159,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T02:49:49.247815+0000","last_change":"2024-09-15T02:49:40.300240+0000","last_active":"2024-09-15T02:49:49.247815+0000","last_peered":"2024-09-15T02:49:49.247815+0000","last_clean":"2024-09-15T02:49:49.247815+0000","last_became_active":"2024-09-15T02:49:18.472004+0000","last_became_peered":"2024-09-15T02:49:18.472004+0000","last_unstale":"2024-09-15T02:49:49.247815+0000","last_undegraded":"2024-09-15T02:49:49.247815+0000","last_fullsized":"2024-09-15T02:49:49.247815+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'89","last_scrub_stamp":"2024-09-15T02:49:31.119567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:37.476646+0000","last_clean_scrub_stamp":"2024-09-15T02:49:31.119567+0000","objects_scrubbed":3,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.244504+0000","scrub_duration":0.61521208699999996,"objects_trimmed":0,"snaptrim_duration":0.011482107,"stat_sum":{"num_bytes":1130496,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,7,5,11,10],"acting":[8,0,3,7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":132825088,"num_objects":121,"num_object_clones":75,"num_object_copies":847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":12,"num_read":269,"num_read_kb":151012,"num_write":528,"num_write_kb":93128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":40222720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250793984,"data_stored":348368896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3956,"internal_metadata":0},"log_size":1729,"ondisk_log_size":1729,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":0,"seq":420906795011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324600,"kb_used_data":23076,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94047240,"statfs":{"total":96636764160,"available":96304373760,"internally_reserved":0,"allocated":23629824,"data_stored":31258234,"data_compressed":155765,"data_compressed_allocated":323584,"data_compressed_original":647168,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.3109999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.425},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":79.933999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":8,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.5269999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":78.307000000000002,"5min":78.307000000000002,"15min":78.307000000000002},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":934.447,"5min":934.447,"15min":934.447},"last":1.458}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":13.593999999999999,"5min":13.593999999999999,"15min":13.593999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":287.02499999999998,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.504},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.482}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.232},{"interface":"front","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.569}]},{"osd":14,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3580000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.02},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":4.2679999999999998,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.619}]}]},{"osd":4,"up_from":22,"seq":94489280531,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323500,"kb_used_data":21980,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94048340,"statfs":{"total":96636764160,"available":96305500160,"internally_reserved":0,"allocated":22507520,"data_stored":32433786,"data_compressed":155765,"data_compressed_allocated":323584,"data_compressed_original":647168,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":50.710000000000001,"5min":50.710000000000001,"15min":50.710000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":743.34100000000001,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.90200000000000002}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":3.4929999999999999,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.218}]},{"osd":5,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":4.141,"5min":4.141,"15min":4.141},"last":0.879}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.177}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":81.322000000000003,"5min":81.322000000000003,"15min":81.322000000000003},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":936.86800000000005,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.379},{"interface":"front","average":{"1min":32.043999999999997,"5min":32.043999999999997,"15min":32.043999999999997},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":586.00900000000001,"5min":586.00900000000001,"15min":586.00900000000001},"last":1.351}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.512},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":1.4059999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.627}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":15,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.153},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.696}]}]},{"osd":5,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315456,"kb_used_data":13940,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056384,"statfs":{"total":96636764160,"available":96313737216,"internally_reserved":0,"allocated":14274560,"data_stored":18444102,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":4,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.3320000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":3.1160000000000001,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.0109999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.452}]},{"osd":11,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":1.111},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.218}]},{"osd":12,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"max":{"1min":3.363,"5min":3.363,"15min":3.363},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":1.4279999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":1.3580000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321592,"kb_used_data":20076,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94050248,"statfs":{"total":96636764160,"available":96307453952,"internally_reserved":0,"allocated":20557824,"data_stored":28782406,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,5,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.758}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":1.248},{"interface":"front","average":{"1min":43.116999999999997,"5min":43.116999999999997,"15min":43.116999999999997},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":503.59300000000002,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.1579999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.607},{"interface":"front","average":{"1min":10.843,"5min":10.843,"15min":10.843},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":223.92500000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.1859999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.6759999999999999},{"interface":"front","average":{"1min":42.957000000000001,"5min":42.957000000000001,"15min":42.957000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":954.81700000000001,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.8140000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":1.855}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8959999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.5680000000000001},{"interface":"front","average":{"1min":29.276,"5min":29.276,"15min":29.276},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":549.39099999999996,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.9770000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":38.726999999999997,"5min":38.726999999999997,"15min":38.726999999999997},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":893.91999999999996,"5min":893.91999999999996,"15min":893.91999999999996},"last":2.0990000000000002},{"interface":"front","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":2.1459999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":6.6509999999999998},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":122.18000000000001,"5min":122.18000000000001,"15min":122.18000000000001},"last":2.1899999999999999},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.5109999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":13.276999999999999,"15min":13.276999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":286.791,"5min":286.791,"15min":286.791},"last":1.542},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.7230000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":22.420999999999999,"5min":22.420999999999999,"15min":22.420999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":504.36000000000001,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.587},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":3.5609999999999999,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.7909999999999999}]}]},{"osd":13,"up_from":22,"seq":94489280532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318748,"kb_used_data":17232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94053092,"statfs":{"total":96636764160,"available":96310366208,"internally_reserved":0,"allocated":17645568,"data_stored":24215944,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.1040000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.214}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.282},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.1859999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.847},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":2.012}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.788},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":1.032,"5min":1.032,"15min":1.032},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1619999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":2.073}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.177},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.5780000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.728,"15min":1.728},"min":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.5},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"min":{"1min":1.512,"5min":1.512,"15min":1.512},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.9410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4609999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"min":{"1min":1.466,"5min":1.466,"15min":1.466},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":2.2130000000000001},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"min":{"1min":1.7,"5min":1.7,"15min":1.7},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":2.4180000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"min":{"1min":1.417,"5min":1.417,"15min":1.417},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":2.129},{"interface":"front","average":{"1min":2,"5min":2,"15min":2},"min":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":2.2959999999999998}]},{"osd":14,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.899,"15min":1.899},"min":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.9950000000000001},{"interface":"front","average":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":3.069,"5min":3.069,"15min":3.069},"last":2.2669999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":38.848999999999997,"5min":38.848999999999997,"15min":38.848999999999997},"min":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"max":{"1min":663.61500000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":2.552},{"interface":"front","average":{"1min":1.861,"5min":1.861,"15min":1.861},"min":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.8240000000000001}]}]},{"osd":1,"up_from":22,"seq":94489280532,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320348,"kb_used_data":18832,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94051492,"statfs":{"total":96636764160,"available":96308727808,"internally_reserved":0,"allocated":19283968,"data_stored":26857864,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":12.202,"5min":12.202,"15min":12.202},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":1.0149999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":0.68000000000000005}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.85599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":4.3129999999999997,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":4.0119999999999996,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":1.181}]},{"osd":9,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.147,"5min":1.147,"15min":1.147},"max":{"1min":4.4809999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.496},{"interface":"front","average":{"1min":1.77,"5min":1.77,"15min":1.77},"min":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"max":{"1min":4.1299999999999999,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.29}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":11,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":12.597,"5min":12.597,"15min":12.597},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":189.79599999999999,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.343},{"interface":"front","average":{"1min":10.946,"5min":10.946,"15min":10.946},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":163.952,"5min":163.952,"15min":163.952},"last":1.0920000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":0.70799999999999996}]},{"osd":14,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":4.4119999999999999,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.385},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"max":{"1min":3.839,"5min":3.839,"15min":3.839},"last":0.90600000000000003}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.53},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.415}]}]},{"osd":14,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316172,"kb_used_data":18816,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94055668,"statfs":{"total":96636764160,"available":96313004032,"internally_reserved":0,"allocated":19267584,"data_stored":26005896,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":1.26},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":1.3500000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":1.083}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.3599999999999999,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":11.234,"5min":11.234,"15min":11.234},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":183.30199999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.3859999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":38.918999999999997,"5min":38.918999999999997,"15min":38.918999999999997},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":681.39599999999996,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.028},{"interface":"front","average":{"1min":52.350000000000001,"5min":52.350000000000001,"15min":52.350000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":921.02099999999996,"5min":921.02099999999996,"15min":921.02099999999996},"last":0.97199999999999998}]},{"osd":5,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.1930000000000001,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":1.119}]},{"osd":9,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.4490000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":11,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":3.133,"5min":3.133,"15min":3.133},"last":1.425},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.6479999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.623},{"interface":"front","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":3.0659999999999998,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.5089999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.6799999999999999},{"interface":"front","average":{"1min":3.367,"5min":3.367,"15min":3.367},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":34.311999999999998,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.909}]}]},{"osd":2,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319180,"kb_used_data":17664,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94052660,"statfs":{"total":96636764160,"available":96309923840,"internally_reserved":0,"allocated":18087936,"data_stored":25916328,"data_compressed":171503,"data_compressed_allocated":372736,"data_compressed_original":1191936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":1.1339999999999999},{"interface":"front","average":{"1min":30.068000000000001,"5min":30.068000000000001,"15min":30.068000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":611.07899999999995,"5min":611.07899999999995,"15min":611.07899999999995},"last":1.569}]},{"osd":1,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":46.866999999999997,"5min":46.866999999999997,"15min":46.866999999999997},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":917.20299999999997,"5min":917.20299999999997,"15min":917.20299999999997},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.635}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.75},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.89400000000000002}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.3440000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.2010000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.5270000000000001,"5min":2.5270000000000001,"15min":2.5270000000000001},"last":1.661},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.7629999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:29 2024","interfaces":[{"interface":"back","average":{"1min":28.021999999999998,"5min":28.021999999999998,"15min":28.021999999999998},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":531.54300000000001,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.448},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":1.8140000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.526},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.3570000000000002,"5min":3.3570000000000002,"15min":3.3570000000000002},"last":1.7310000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316120,"kb_used_data":14604,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94055720,"statfs":{"total":96636764160,"available":96313057280,"internally_reserved":0,"allocated":14954496,"data_stored":20689288,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":1.179}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.0069999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.165},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.488}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.679}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5860000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:49:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":1.704},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":1.413}]},{"osd":12,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":6.4160000000000004,"5min":6.4160000000000004,"15min":6.4160000000000004},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":121.304,"5min":121.304,"15min":121.304},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.45,"15min":1.45},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.363}]},{"osd":13,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5979999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":1.093,"5min":1.093,"15min":1.093},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.4590000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":1.26,"5min":1.26,"15min":1.26},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.512},{"interface":"front","average":{"1min":18.201000000000001,"5min":18.201000000000001,"15min":18.201000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":383.084,"5min":383.084,"15min":383.084},"last":1.611}]}]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313900,"kb_used_data":16544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94057940,"statfs":{"total":96636764160,"available":96315330560,"internally_reserved":0,"allocated":16941056,"data_stored":23196040,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.91400000000000003},{"interface":"front","average":{"1min":42.505000000000003,"5min":42.505000000000003,"15min":42.505000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":913.93899999999996,"5min":913.93899999999996,"15min":913.93899999999996},"last":0.67500000000000004}]},{"osd":4,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.7070000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.329}]},{"osd":12,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":1.004},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":1.024,"5min":1.024,"15min":1.024},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.464}]},{"osd":13,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.117,"5min":1.117,"15min":1.117},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":0.97499999999999998},{"interface":"front","average":{"1min":5.4930000000000003,"5min":5.4930000000000003,"15min":5.4930000000000003},"min":{"1min":1.153,"5min":1.153,"15min":1.153},"max":{"1min":92.698999999999998,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.3109999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]}]},{"osd":8,"up_from":22,"seq":94489280532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309604,"kb_used_data":12248,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94062236,"statfs":{"total":96636764160,"available":96319729664,"internally_reserved":0,"allocated":12541952,"data_stored":16744859,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":22.859000000000002,"5min":22.859000000000002,"15min":22.859000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":382.21199999999999,"5min":382.21199999999999,"15min":382.21199999999999},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.0289999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.80000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":10.6,"5min":10.6,"15min":10.6},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":182.642,"5min":182.642,"15min":182.642},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.68200000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.879},{"interface":"front","average":{"1min":43.805,"5min":43.805,"15min":43.805},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":849.11599999999999,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.2509999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":0.93400000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":39.085999999999999,"5min":39.085999999999999,"15min":39.085999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":423.70800000000003,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.093},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.1240000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":1.103,"5min":1.103,"15min":1.103},"max":{"1min":3.012,"5min":3.012,"15min":3.012},"last":1.298}]},{"osd":15,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":33.267000000000003,"5min":33.267000000000003,"15min":33.267000000000003},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":603.77599999999995,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.28}]}]},{"osd":9,"up_from":22,"seq":94489280532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307748,"kb_used_data":10392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94064092,"statfs":{"total":96636764160,"available":96321630208,"internally_reserved":0,"allocated":10641408,"data_stored":12144473,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.85999999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.262}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.92500000000000004}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.1910000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.3819999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.113},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.361}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.619,"15min":1.619},"min":{"1min":1.002,"5min":1.002,"15min":1.002},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.347,"5min":2.347,"15min":2.347},"last":1.4890000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626447,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308064,"kb_used_data":10184,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94063776,"statfs":{"total":96636764160,"available":96321306624,"internally_reserved":0,"allocated":10428416,"data_stored":12501384,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280533,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325016,"kb_used_data":23500,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94046824,"statfs":{"total":96636764160,"available":96303947776,"internally_reserved":0,"allocated":24064000,"data_stored":33915816,"data_compressed":171503,"data_compressed_allocated":372736,"data_compressed_original":1191936,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":1.077},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.0469999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":0.878},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.94899999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.496,"5min":2.496,"15min":2.496},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":4.1559999999999997,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.292}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.1919999999999999},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.113}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":1.341},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.556}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.7010000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.667},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.5009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.613},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.5309999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":31.167999999999999,"5min":31.167999999999999,"15min":31.167999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":713.26999999999998,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.585}]},{"osd":13,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":28.263000000000002,"5min":28.263000000000002,"15min":28.263000000000002},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":637.95699999999999,"5min":637.95699999999999,"15min":637.95699999999999},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.863}]},{"osd":14,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.7,"15min":1.7},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.976,"5min":1.976,"15min":1.976},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":5.0839999999999996,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.6359999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":52.142000000000003,"5min":52.142000000000003,"15min":52.142000000000003},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":586.40499999999997,"5min":586.40499999999997,"15min":586.40499999999997},"last":1.837},{"interface":"front","average":{"1min":1.631,"5min":1.631,"15min":1.631},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.8559999999999999,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.7270000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309920,"kb_used_data":12564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94061920,"statfs":{"total":96636764160,"available":96319406080,"internally_reserved":0,"allocated":12865536,"data_stored":16519560,"data_compressed":163898,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.1040000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.603},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.6699999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.474}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.72},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.75}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.6459999999999999},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.8520000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.825}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":3.7440000000000002,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.365}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.887},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.9119999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0529999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1549999999999998}]},{"osd":11,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":2.0939999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":2.319}]},{"osd":13,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":2.121},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":2.2480000000000002}]},{"osd":14,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":3.819,"5min":3.819,"15min":3.819},"last":2.6120000000000001},{"interface":"front","average":{"1min":15.798999999999999,"5min":15.798999999999999,"15min":15.798999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":285.95499999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":2.0230000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19345408,"data_stored":27873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18067456,"data_stored":25948160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16818176,"data_stored":24477696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15724544,"data_stored":22286336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21340160,"data_stored":31563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13062144,"data_stored":17534976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11354112,"data_stored":13365248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3956,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":22462464,"data_stored":30388224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11321344,"data_stored":15835136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9424896,"data_stored":11235328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9211904,"data_stored":11591680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":22794240,"data_stored":32477184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11649024,"data_stored":15609856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":16429056,"data_stored":23306240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":18051072,"data_stored":25096192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13737984,"data_stored":19779584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:49:53.378 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-15T02:49:53.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-15T02:49:53.388 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:49:53.418 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:53.411+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:53.466 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-15T02:49:53.475 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:53.475 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:53.484 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:49:53.494 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-15T02:49:53.520 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-15T02:49:53.519+0000 7fe7158fc700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 22072) UID: 0 2024-09-15T02:49:53.520 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-15T02:49:53.552 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-15T02:49:53.574 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:53.575 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:53.583 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-15T02:49:53.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T02:49:53.611 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-15T02:49:53.621 INFO:tasks.ceph.osd.11.smithi191.stderr:2024-09-15T02:49:53.620+0000 7f53f13b5700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 22450) UID: 0 2024-09-15T02:49:53.635 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-15T02:49:53.663 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-15T02:49:53.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:49:53.663+0000 7fcb677fe700 0 -- 172.21.15.6:0/1265439396 >> [v2:172.21.15.152:6808/22077,v1:172.21.15.152:6811/22077] conn(0x7fcb68192fa0 msgr2=0x7fcb68195380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T02:49:53.682 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:53.682 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:53.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T02:49:53.700 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-15T02:49:53.722 INFO:tasks.ceph.osd.14.smithi152.stderr:2024-09-15T02:49:53.718+0000 7f91a78c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 22072) UID: 0 2024-09-15T02:49:53.727 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T02:49:53.729 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T02:49:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T02:49:53.799 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:53.799 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:53.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T02:49:53.822 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:53.819+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:53.903 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:53.903 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:53.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T02:49:53.923 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-15T02:49:53.922+0000 7f2c40a4b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22066) UID: 0 2024-09-15T02:49:54.011 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.011 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T02:49:54.023 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-15T02:49:54.019+0000 7f3be7d7b700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22071) UID: 0 2024-09-15T02:49:54.117 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.117 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.124 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-15T02:49:54.119+0000 7f2388956700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22123) UID: 0 2024-09-15T02:49:54.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T02:49:54.225 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-15T02:49:54.219+0000 7ff1aa05d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 22128) UID: 0 2024-09-15T02:49:54.226 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.226 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T02:49:54.250 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_0' removed 2024-09-15T02:49:54.265 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-15T02:49:54.274 INFO:tasks.thrashosds:joining thrashosds 2024-09-15T02:49:54.344 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.344 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T02:49:54.452 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.452 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T02:49:54.558 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.558 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T02:49:54.665 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.665 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T02:49:54.772 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.772 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T02:49:54.894 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:54.894 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:54.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T02:49:55.006 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.006 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T02:49:55.112 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.113 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.121 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T02:49:55.218 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.218 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.227 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T02:49:55.326 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.327 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T02:49:55.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T02:49:55.429 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.430 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.439 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T02:49:55.539 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.539 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T02:49:55.644 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.644 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.654 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T02:49:55.749 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.750 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T02:49:55.857 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.857 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T02:49:55.963 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:55.963 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:55.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T02:49:56.069 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.070 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T02:49:56.177 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.178 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.187 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T02:49:56.295 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.295 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.305 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T02:49:56.401 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.401 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.411 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T02:49:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-15T02:49:56.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T02:49:56.506 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.507 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.516 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T02:49:56.614 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.614 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.623 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T02:49:56.721 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.721 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.730 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T02:49:56.826 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.826 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T02:49:56.947 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:56.947 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:56.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T02:49:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.060 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T02:49:57.168 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.168 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T02:49:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T02:49:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-15T02:49:57.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T02:49:57.426 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-15T02:49:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T02:49:57.550 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a: {} 2024-09-15T02:49:57.573 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b: {} 2024-09-15T02:49:57.595 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c: {} 2024-09-15T02:49:57.597 INFO:teuthology.orchestra.run.smithi006.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T02:49:57.597 INFO:teuthology.orchestra.run.smithi006.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T02:49:57.598 INFO:teuthology.orchestra.run.smithi006.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T02:49:57.611 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.611 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-15T02:49:57.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T02:49:57.717 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.718 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T02:49:57.827 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.827 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T02:49:57.937 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:57.937 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:57.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T02:49:58.042 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.042 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T02:49:58.152 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.152 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.162 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T02:49:58.273 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.273 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.283 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T02:49:58.383 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.383 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.392 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T02:49:58.429 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.0 to 1 (10000) 2024-09-15T02:49:58.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-15T02:49:58.494 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.494 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.503 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T02:49:58.598 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.599 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.608 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T02:49:58.703 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.703 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:58.712 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T02:49:58.808 INFO:teuthology.orchestra.run.smithi152.stdout:ERROR: (22) Invalid argument 2024-09-15T02:49:58.808 INFO:teuthology.orchestra.run.smithi152.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T02:49:59.445 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-15T02:49:59.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-15T02:50:00.447 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.4 to 1 (10000) 2024-09-15T02:50:00.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-15T02:50:01.461 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-15T02:50:01.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-15T02:50:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.8 to 1 (10000) 2024-09-15T02:50:02.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-15T02:50:03.476 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-15T02:50:03.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-15T02:50:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.12 to 1 (10000) 2024-09-15T02:50:04.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-15T02:50:04.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-15T02:50:04.819+0000 7f69ef7fe700 0 --2- 172.21.15.6:0/1770909199 >> v2:172.21.15.118:3300/0 conn(0x7f69f014bff0 0x7f69f0155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T02:50:05.512 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-15T02:50:05.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-15T02:50:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.1 to 1 (10000) 2024-09-15T02:50:06.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-15T02:50:07.524 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-15T02:50:07.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-15T02:50:08.542 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.5 to 1 (10000) 2024-09-15T02:50:08.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-15T02:50:09.545 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-15T02:50:09.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-15T02:50:10.555 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.9 to 1 (10000) 2024-09-15T02:50:10.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-15T02:50:11.564 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-15T02:50:11.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-15T02:50:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.13 to 1 (10000) 2024-09-15T02:50:12.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-15T02:50:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-15T02:50:13.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-15T02:50:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.2 to 1 (10000) 2024-09-15T02:50:14.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-15T02:50:15.610 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-15T02:50:15.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-15T02:50:16.615 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.14 to 1 (10000) 2024-09-15T02:50:16.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-15T02:50:17.629 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-15T02:50:17.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-15T02:50:18.633 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.3 to 1 (10000) 2024-09-15T02:50:18.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-15T02:50:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-15T02:50:19.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-15T02:50:20.650 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.7 to 1 (10000) 2024-09-15T02:50:20.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-15T02:50:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-15T02:50:21.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-15T02:50:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.11 to 1 (10000) 2024-09-15T02:50:22.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-15T02:50:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-15T02:50:23.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-15T02:50:24.692 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.15 to 1 (10000) 2024-09-15T02:50:24.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-15T02:50:25.702 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-15T02:50:25.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-15T02:50:26.711 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.10 to 1 (10000) 2024-09-15T02:50:26.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-15T02:50:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-15T02:50:27.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-15T02:50:28.731 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.6 to 1 (10000) 2024-09-15T02:50:28.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-15T02:50:29.740 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-15T02:50:29.755 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-15T02:50:29.755 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-15T02:50:29.756 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-15T02:50:29.756 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-15T02:50:29.756 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T02:50:29.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:50:30.069 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:30.069 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":142,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:50:29.725969+0000","last_up_change":"2024-09-15T02:49:42.289183+0000","last_in_change":"2024-09-15T02:49:52.362323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":74,"up_from":98,"up_thru":106,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":34477},{"type":"v1","addr":"172.21.15.152:6801","nonce":34477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":34477},{"type":"v1","addr":"172.21.15.152:6803","nonce":34477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":34477},{"type":"v1","addr":"172.21.15.152:6807","nonce":34477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":34477},{"type":"v1","addr":"172.21.15.152:6805","nonce":34477}]},"public_addr":"172.21.15.152:6801/34477","cluster_addr":"172.21.15.152:6803/34477","heartbeat_back_addr":"172.21.15.152:6807/34477","heartbeat_front_addr":"172.21.15.152:6805/34477","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":29,"up_from":42,"up_thru":76,"down_at":31,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":27185},{"type":"v1","addr":"172.21.15.152:6812","nonce":27185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":27185},{"type":"v1","addr":"172.21.15.152:6819","nonce":27185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":27185},{"type":"v1","addr":"172.21.15.152:6831","nonce":27185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":27185},{"type":"v1","addr":"172.21.15.152:6826","nonce":27185}]},"public_addr":"172.21.15.152:6812/27185","cluster_addr":"172.21.15.152:6819/27185","heartbeat_back_addr":"172.21.15.152:6831/27185","heartbeat_front_addr":"172.21.15.152:6826/27185","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:47.242372+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:34.269618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:39.185164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.093348+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.890794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:43.303875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:41.805268+0000","dead_epoch":75},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.289669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:36.555828+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:44.408648+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:29.360419+0000","dead_epoch":31},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.224457+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:48.195616+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.117214+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:28.670589+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:50.102496+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:50:30.083 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T02:50:30.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:50:30.380 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:30.380 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":142,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:50:29.725969+0000","last_up_change":"2024-09-15T02:49:42.289183+0000","last_in_change":"2024-09-15T02:49:52.362323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":74,"up_from":98,"up_thru":106,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":34477},{"type":"v1","addr":"172.21.15.152:6801","nonce":34477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":34477},{"type":"v1","addr":"172.21.15.152:6803","nonce":34477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":34477},{"type":"v1","addr":"172.21.15.152:6807","nonce":34477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":34477},{"type":"v1","addr":"172.21.15.152:6805","nonce":34477}]},"public_addr":"172.21.15.152:6801/34477","cluster_addr":"172.21.15.152:6803/34477","heartbeat_back_addr":"172.21.15.152:6807/34477","heartbeat_front_addr":"172.21.15.152:6805/34477","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":29,"up_from":42,"up_thru":76,"down_at":31,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":27185},{"type":"v1","addr":"172.21.15.152:6812","nonce":27185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":27185},{"type":"v1","addr":"172.21.15.152:6819","nonce":27185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":27185},{"type":"v1","addr":"172.21.15.152:6831","nonce":27185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":27185},{"type":"v1","addr":"172.21.15.152:6826","nonce":27185}]},"public_addr":"172.21.15.152:6812/27185","cluster_addr":"172.21.15.152:6819/27185","heartbeat_back_addr":"172.21.15.152:6831/27185","heartbeat_front_addr":"172.21.15.152:6826/27185","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:47.242372+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:34.269618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:39.185164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.093348+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.890794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:43.303875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:41.805268+0000","dead_epoch":75},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.289669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:36.555828+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:44.408648+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:29.360419+0000","dead_epoch":31},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.224457+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:48.195616+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.117214+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:28.670589+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:50.102496+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:50:30.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T02:50:30.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T02:50:30.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T02:50:30.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T02:50:30.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T02:50:30.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T02:50:30.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T02:50:30.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T02:50:30.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-15T02:50:30.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-15T02:50:30.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-15T02:50:30.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-15T02:50:30.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-15T02:50:30.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-15T02:50:30.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-15T02:50:30.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-15T02:50:30.747 INFO:teuthology.orchestra.run.smithi006.stdout:420906795019 2024-09-15T02:50:30.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T02:50:30.755 INFO:teuthology.orchestra.run.smithi006.stdout:180388626455 2024-09-15T02:50:30.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T02:50:30.761 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T02:50:30.766 INFO:teuthology.orchestra.run.smithi006.stdout:94489280541 2024-09-15T02:50:30.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T02:50:30.767 INFO:teuthology.orchestra.run.smithi006.stdout:90194313245 2024-09-15T02:50:30.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T02:50:30.777 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-15T02:50:30.807 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T02:50:30.821 INFO:teuthology.orchestra.run.smithi006.stdout:94489280542 2024-09-15T02:50:30.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T02:50:30.831 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.832 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T02:50:30.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-15T02:50:30.837 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T02:50:30.840 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-15T02:50:30.842 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T02:50:30.849 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T02:50:30.853 INFO:teuthology.orchestra.run.smithi006.stdout:94489280541 2024-09-15T02:50:30.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T02:50:30.905 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:30.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-15T02:50:31.601 INFO:teuthology.orchestra.run.smithi006.stdout:94489280539 2024-09-15T02:50:31.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.14 2024-09-15T02:50:31.642 INFO:teuthology.orchestra.run.smithi006.stdout:94489280539 2024-09-15T02:50:31.667 INFO:teuthology.orchestra.run.smithi006.stdout:94489280539 2024-09-15T02:50:31.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.13 2024-09-15T02:50:31.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.8 2024-09-15T02:50:31.696 INFO:teuthology.orchestra.run.smithi006.stdout:94489280539 2024-09-15T02:50:31.699 INFO:teuthology.orchestra.run.smithi006.stdout:94489280539 2024-09-15T02:50:31.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.4 2024-09-15T02:50:31.727 INFO:teuthology.orchestra.run.smithi006.stdout:90194313244 2024-09-15T02:50:31.728 INFO:teuthology.orchestra.run.smithi006.stdout:420906795018 2024-09-15T02:50:31.731 INFO:teuthology.orchestra.run.smithi006.stdout:94489280541 2024-09-15T02:50:31.733 INFO:teuthology.orchestra.run.smithi006.stdout:180388626454 2024-09-15T02:50:31.734 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.0 2024-09-15T02:50:31.738 INFO:teuthology.orchestra.run.smithi006.stdout:94489280539 2024-09-15T02:50:31.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313244 for osd.5 2024-09-15T02:50:31.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280541 for osd.7 2024-09-15T02:50:31.755 INFO:teuthology.orchestra.run.smithi006.stdout:94489280541 2024-09-15T02:50:31.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 420906795019 got 420906795018 for osd.6 2024-09-15T02:50:31.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 180388626455 got 180388626454 for osd.10 2024-09-15T02:50:31.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.1 2024-09-15T02:50:31.771 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:31.776 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:31.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.3 2024-09-15T02:50:31.781 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:31.784 INFO:teuthology.orchestra.run.smithi006.stdout:94489280541 2024-09-15T02:50:31.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.2 2024-09-15T02:50:31.787 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:31.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.12 2024-09-15T02:50:31.796 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:31.797 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.11 2024-09-15T02:50:31.797 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:31.803 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:31.804 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:31.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.15 2024-09-15T02:50:31.816 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:31.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.9 2024-09-15T02:50:31.816 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:32.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-15T02:50:32.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-15T02:50:32.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T02:50:32.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T02:50:32.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T02:50:32.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T02:50:32.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T02:50:32.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T02:50:32.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T02:50:32.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T02:50:33.171 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:33.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.14 2024-09-15T02:50:33.209 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.221 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:33.224 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:33.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.13 2024-09-15T02:50:33.241 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.0 2024-09-15T02:50:33.261 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.281 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:33.282 INFO:teuthology.orchestra.run.smithi006.stdout:94489280542 2024-09-15T02:50:33.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.8 2024-09-15T02:50:33.295 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.299 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.7 2024-09-15T02:50:33.299 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.308 INFO:teuthology.orchestra.run.smithi006.stdout:90194313245 2024-09-15T02:50:33.323 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313245 for osd.5 2024-09-15T02:50:33.323 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.332 INFO:teuthology.orchestra.run.smithi006.stdout:420906795019 2024-09-15T02:50:33.333 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:33.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 420906795019 got 420906795019 for osd.6 2024-09-15T02:50:33.346 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.346 INFO:teuthology.orchestra.run.smithi006.stdout:94489280540 2024-09-15T02:50:33.351 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.4 2024-09-15T02:50:33.351 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280540 for osd.1 2024-09-15T02:50:33.358 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.395 INFO:teuthology.orchestra.run.smithi006.stdout:180388626455 2024-09-15T02:50:33.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 180388626455 got 180388626455 for osd.10 2024-09-15T02:50:33.408 DEBUG:teuthology.parallel:result is None 2024-09-15T02:50:33.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-15T02:50:33.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:50:33.756 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:33.759 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:50:33.770 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T02:50:32.989186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4835588,"kb_used_data":26260,"kb_used_omap":36,"kb_used_meta":4809051,"kb_avail":1505113852,"statfs":{"total":1546188226560,"available":1541236584448,"internally_reserved":0,"allocated":26890240,"data_stored":21243372,"data_compressed":3822445,"data_compressed_allocated":7925760,"data_compressed_original":16744448,"omap_allocated":37677,"internal_metadata":4924468435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.486502+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:59.318029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:59.318029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:26.106992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.955225+0000","last_change":"2024-09-15T02:49:53.389280+0000","last_active":"2024-09-15T02:50:28.955225+0000","last_peered":"2024-09-15T02:50:28.955225+0000","last_clean":"2024-09-15T02:50:28.955225+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:28.955225+0000","last_undegraded":"2024-09-15T02:50:28.955225+0000","last_fullsized":"2024-09-15T02:50:28.955225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.695063+0000","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.154656+0000","last_change":"2024-09-15T02:49:53.385816+0000","last_active":"2024-09-15T02:50:29.154656+0000","last_peered":"2024-09-15T02:50:29.154656+0000","last_clean":"2024-09-15T02:50:29.154656+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:29.154656+0000","last_undegraded":"2024-09-15T02:50:29.154656+0000","last_fullsized":"2024-09-15T02:50:29.154656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:08.412981+0000","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.060075+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:50:29.060075+0000","last_peered":"2024-09-15T02:50:29.060075+0000","last_clean":"2024-09-15T02:50:29.060075+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:29.060075+0000","last_undegraded":"2024-09-15T02:50:29.060075+0000","last_fullsized":"2024-09-15T02:50:29.060075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":64,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953455+0000","last_change":"2024-09-15T02:50:04.489301+0000","last_active":"2024-09-15T02:50:28.953455+0000","last_peered":"2024-09-15T02:50:28.953455+0000","last_clean":"2024-09-15T02:50:28.953455+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:28.953455+0000","last_undegraded":"2024-09-15T02:50:28.953455+0000","last_fullsized":"2024-09-15T02:50:28.953455+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T02:49:56.345917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:56.345917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.722217+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:50:28.722217+0000","last_peered":"2024-09-15T02:50:28.722217+0000","last_clean":"2024-09-15T02:50:28.722217+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:28.722217+0000","last_undegraded":"2024-09-15T02:50:28.722217+0000","last_fullsized":"2024-09-15T02:50:28.722217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.613224+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.853218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.853218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.808351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954684+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:50:28.954684+0000","last_peered":"2024-09-15T02:50:28.954684+0000","last_clean":"2024-09-15T02:50:28.954684+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:28.954684+0000","last_undegraded":"2024-09-15T02:50:28.954684+0000","last_fullsized":"2024-09-15T02:50:28.954684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954732+0000","last_change":"2024-09-15T02:50:03.139015+0000","last_active":"2024-09-15T02:50:28.954732+0000","last_peered":"2024-09-15T02:50:28.954732+0000","last_clean":"2024-09-15T02:50:28.954732+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:28.954732+0000","last_undegraded":"2024-09-15T02:50:28.954732+0000","last_fullsized":"2024-09-15T02:50:28.954732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-15T02:50:03.138800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:50:03.138800+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.267089+0000","scrub_duration":0.0046469529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300144,"kb_used_data":1632,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071696,"statfs":{"total":96636764160,"available":96329416704,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]}]},{"osd":7,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303140,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068700,"statfs":{"total":96636764160,"available":96326348800,"internally_reserved":0,"allocated":1662976,"data_stored":1252855,"data_compressed":236080,"data_compressed_allocated":487424,"data_compressed_original":974848,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":523.76099999999997}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.101}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.7270000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.1919999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.302},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.7749999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.641},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.246}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.599},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.6890000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.45},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.7609999999999999}]}]},{"osd":4,"up_from":22,"seq":94489280540,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.96499999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.2809999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":1.377},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.2450000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.4419999999999999},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":1.321}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.554},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":1.4730000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.6519999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.679},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.5780000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.6100000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.522},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.696}]}]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3759999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.5509999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.7689999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.6830000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.73},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.857}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":50.643000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":2.04}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.68600000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.052},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.0049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.4730000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.73699999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.264},{"interface":"front","average":{"1min":29.276,"5min":29.276,"15min":29.276},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":549.39099999999996,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.413}]},{"osd":12,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":38.726999999999997,"5min":38.726999999999997,"15min":38.726999999999997},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":893.91999999999996,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.579},{"interface":"front","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.6060000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":6.6509999999999998},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":122.18000000000001,"5min":122.18000000000001,"15min":122.18000000000001},"last":1.538},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.2869999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":13.276999999999999,"15min":13.276999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":286.791,"5min":286.791,"15min":286.791},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.3600000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.4570000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":1.0149999999999999},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.728,"15min":1.728},"min":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"min":{"1min":1.512,"5min":1.512,"15min":1.512},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.4399999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"min":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6120000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":27.055,"5min":27.055,"15min":27.055},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":501.60599999999999,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.411},{"interface":"front","average":{"1min":2.157,"5min":2.157,"15min":2.157},"min":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.5089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"min":{"1min":1.466,"5min":1.466,"15min":1.466},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.472},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"min":{"1min":1.7,"5min":1.7,"15min":1.7},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5840000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.357},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.659}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.5580000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.776},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.681}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.365}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.028}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":0.91100000000000003}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.4850000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.3380000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":12.597,"5min":12.597,"15min":12.597},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":189.79599999999999,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":10.946,"5min":10.946,"15min":10.946},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":163.952,"5min":163.952,"15min":163.952},"last":1.129}]},{"osd":13,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.244}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.5189999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.55}]}]},{"osd":14,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":1.24},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":0.92900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":1.095},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.131},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.175}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":0.98999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":1.1639999999999999},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.3720000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.331}]},{"osd":10,"last update":"Sun Sep 15 02:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.756}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":2.0009999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9550000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.7310000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.79500000000000004},{"interface":"front","average":{"1min":30.068000000000001,"5min":30.068000000000001,"15min":30.068000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":611.07899999999995,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.91500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.024}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.258},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.0620000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.179},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.232}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.399},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":1.3240000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.54},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.518}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.29},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.482}]}]},{"osd":15,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":0.72299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.111},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2729999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.3}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3320000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":2.069}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.3839999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.413}]}]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":1.0249999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":1.222},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.329}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.2829999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.2609999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.165},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.3999999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303160,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068680,"statfs":{"total":96636764160,"available":96326328320,"internally_reserved":0,"allocated":1679360,"data_stored":1262348,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":22.859000000000002,"5min":22.859000000000002,"15min":22.859000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":382.21199999999999,"5min":382.21199999999999,"15min":382.21199999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.109},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.3540000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.056},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.486}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":1.579},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.613}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.159},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.6379999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.778},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.2969999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.661},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.71}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.821}]}]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1261770,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":3.8540000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.71499999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.3180000000000001},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":0.88600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.0009999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.0489999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4019999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5069999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.7050000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":4.2119999999999997},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.3640000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.5449999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299504,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072336,"statfs":{"total":96636764160,"available":96330072064,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90600000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":12.095000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":189.78700000000001,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.101},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.165}]},{"osd":4,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":4.0759999999999996,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.409},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.1319999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.238},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2150000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.4410000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.4970000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":9.3740000000000006,"5min":9.3740000000000006,"15min":9.3740000000000006},"last":1.556},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.6220000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":11.145,"5min":11.145,"15min":11.145},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":170.46600000000001,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":17.238,"5min":17.238,"15min":17.238},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":272.40899999999999,"5min":272.40899999999999,"15min":272.40899999999999},"last":1.423}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.585},{"interface":"front","average":{"1min":35.201000000000001,"5min":35.201000000000001,"15min":35.201000000000001},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":603.03999999999996,"5min":603.03999999999996,"15min":603.03999999999996},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.081}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.504},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.5600000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.135},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.625}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.698},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.736}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.796},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.655}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.829},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.7589999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":2.0299999999999998},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.972}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.929}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":2.0760000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.996}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.123}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.874},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.3340000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.431},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.169}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.294},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.202}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.2509999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.5229999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.3320000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.383},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":1.6060000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:50:33.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:50:34.068 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:50:34.083 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T02:50:32.989186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4835588,"kb_used_data":26260,"kb_used_omap":36,"kb_used_meta":4809051,"kb_avail":1505113852,"statfs":{"total":1546188226560,"available":1541236584448,"internally_reserved":0,"allocated":26890240,"data_stored":21243372,"data_compressed":3822445,"data_compressed_allocated":7925760,"data_compressed_original":16744448,"omap_allocated":37677,"internal_metadata":4924468435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.486502+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:59.318029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:59.318029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:26.106992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.955225+0000","last_change":"2024-09-15T02:49:53.389280+0000","last_active":"2024-09-15T02:50:28.955225+0000","last_peered":"2024-09-15T02:50:28.955225+0000","last_clean":"2024-09-15T02:50:28.955225+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:28.955225+0000","last_undegraded":"2024-09-15T02:50:28.955225+0000","last_fullsized":"2024-09-15T02:50:28.955225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.695063+0000","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.154656+0000","last_change":"2024-09-15T02:49:53.385816+0000","last_active":"2024-09-15T02:50:29.154656+0000","last_peered":"2024-09-15T02:50:29.154656+0000","last_clean":"2024-09-15T02:50:29.154656+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:29.154656+0000","last_undegraded":"2024-09-15T02:50:29.154656+0000","last_fullsized":"2024-09-15T02:50:29.154656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:08.412981+0000","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.060075+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:50:29.060075+0000","last_peered":"2024-09-15T02:50:29.060075+0000","last_clean":"2024-09-15T02:50:29.060075+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:29.060075+0000","last_undegraded":"2024-09-15T02:50:29.060075+0000","last_fullsized":"2024-09-15T02:50:29.060075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":64,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953455+0000","last_change":"2024-09-15T02:50:04.489301+0000","last_active":"2024-09-15T02:50:28.953455+0000","last_peered":"2024-09-15T02:50:28.953455+0000","last_clean":"2024-09-15T02:50:28.953455+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:28.953455+0000","last_undegraded":"2024-09-15T02:50:28.953455+0000","last_fullsized":"2024-09-15T02:50:28.953455+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T02:49:56.345917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:56.345917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.722217+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:50:28.722217+0000","last_peered":"2024-09-15T02:50:28.722217+0000","last_clean":"2024-09-15T02:50:28.722217+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:28.722217+0000","last_undegraded":"2024-09-15T02:50:28.722217+0000","last_fullsized":"2024-09-15T02:50:28.722217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.613224+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.853218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.853218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.808351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954684+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:50:28.954684+0000","last_peered":"2024-09-15T02:50:28.954684+0000","last_clean":"2024-09-15T02:50:28.954684+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:28.954684+0000","last_undegraded":"2024-09-15T02:50:28.954684+0000","last_fullsized":"2024-09-15T02:50:28.954684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954732+0000","last_change":"2024-09-15T02:50:03.139015+0000","last_active":"2024-09-15T02:50:28.954732+0000","last_peered":"2024-09-15T02:50:28.954732+0000","last_clean":"2024-09-15T02:50:28.954732+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:28.954732+0000","last_undegraded":"2024-09-15T02:50:28.954732+0000","last_fullsized":"2024-09-15T02:50:28.954732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-15T02:50:03.138800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:50:03.138800+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.267089+0000","scrub_duration":0.0046469529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300144,"kb_used_data":1632,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071696,"statfs":{"total":96636764160,"available":96329416704,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]}]},{"osd":7,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303140,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068700,"statfs":{"total":96636764160,"available":96326348800,"internally_reserved":0,"allocated":1662976,"data_stored":1252855,"data_compressed":236080,"data_compressed_allocated":487424,"data_compressed_original":974848,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":523.76099999999997}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.101}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.7270000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.1919999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.302},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.7749999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.641},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.246}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.599},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.6890000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.45},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.7609999999999999}]}]},{"osd":4,"up_from":22,"seq":94489280540,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.96499999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.2809999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":1.377},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.2450000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.4419999999999999},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":1.321}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.554},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":1.4730000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.6519999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.679},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.5780000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.6100000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.522},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.696}]}]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3759999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.5509999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.7689999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.6830000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.73},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.857}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":50.643000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":2.04}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.68600000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.052},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.0049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.4730000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.73699999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.264},{"interface":"front","average":{"1min":29.276,"5min":29.276,"15min":29.276},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":549.39099999999996,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.413}]},{"osd":12,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":38.726999999999997,"5min":38.726999999999997,"15min":38.726999999999997},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":893.91999999999996,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.579},{"interface":"front","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.6060000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":6.6509999999999998},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":122.18000000000001,"5min":122.18000000000001,"15min":122.18000000000001},"last":1.538},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.2869999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":13.276999999999999,"15min":13.276999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":286.791,"5min":286.791,"15min":286.791},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.3600000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.4570000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":1.0149999999999999},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.728,"15min":1.728},"min":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"min":{"1min":1.512,"5min":1.512,"15min":1.512},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.4399999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"min":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6120000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":27.055,"5min":27.055,"15min":27.055},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":501.60599999999999,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.411},{"interface":"front","average":{"1min":2.157,"5min":2.157,"15min":2.157},"min":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.5089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"min":{"1min":1.466,"5min":1.466,"15min":1.466},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.472},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"min":{"1min":1.7,"5min":1.7,"15min":1.7},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5840000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.357},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.659}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.5580000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.776},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.681}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.365}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.028}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":0.91100000000000003}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.4850000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.3380000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":12.597,"5min":12.597,"15min":12.597},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":189.79599999999999,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":10.946,"5min":10.946,"15min":10.946},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":163.952,"5min":163.952,"15min":163.952},"last":1.129}]},{"osd":13,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.244}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.5189999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.55}]}]},{"osd":14,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":1.24},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":0.92900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":1.095},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.131},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.175}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":0.98999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":1.1639999999999999},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.3720000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.331}]},{"osd":10,"last update":"Sun Sep 15 02:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.756}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":2.0009999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9550000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.7310000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.79500000000000004},{"interface":"front","average":{"1min":30.068000000000001,"5min":30.068000000000001,"15min":30.068000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":611.07899999999995,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.91500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.024}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.258},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.0620000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.179},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.232}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.399},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":1.3240000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.54},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.518}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.29},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.482}]}]},{"osd":15,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":0.72299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.111},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2729999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.3}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3320000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":2.069}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.3839999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.413}]}]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":1.0249999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":1.222},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.329}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.2829999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.2609999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.165},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.3999999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303160,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068680,"statfs":{"total":96636764160,"available":96326328320,"internally_reserved":0,"allocated":1679360,"data_stored":1262348,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":22.859000000000002,"5min":22.859000000000002,"15min":22.859000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":382.21199999999999,"5min":382.21199999999999,"15min":382.21199999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.109},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.3540000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.056},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.486}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":1.579},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.613}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.159},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.6379999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.778},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.2969999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.661},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.71}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.821}]}]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1261770,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":3.8540000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.71499999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.3180000000000001},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":0.88600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.0009999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.0489999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4019999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5069999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.7050000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":4.2119999999999997},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.3640000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.5449999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299504,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072336,"statfs":{"total":96636764160,"available":96330072064,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90600000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":12.095000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":189.78700000000001,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.101},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.165}]},{"osd":4,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":4.0759999999999996,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.409},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.1319999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.238},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2150000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.4410000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.4970000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":9.3740000000000006,"5min":9.3740000000000006,"15min":9.3740000000000006},"last":1.556},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.6220000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":11.145,"5min":11.145,"15min":11.145},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":170.46600000000001,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":17.238,"5min":17.238,"15min":17.238},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":272.40899999999999,"5min":272.40899999999999,"15min":272.40899999999999},"last":1.423}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.585},{"interface":"front","average":{"1min":35.201000000000001,"5min":35.201000000000001,"15min":35.201000000000001},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":603.03999999999996,"5min":603.03999999999996,"15min":603.03999999999996},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.081}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.504},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.5600000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.135},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.625}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.698},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.736}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.796},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.655}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.829},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.7589999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":2.0299999999999998},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.972}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.929}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":2.0760000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.996}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.123}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.874},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.3340000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.431},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.169}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.294},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.202}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.2509999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.5229999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.3320000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.383},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":1.6060000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:50:34.141 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-15T02:50:34.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-15T02:50:34.462 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T02:50:34.470 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T02:50:34.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:50:34.758 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:34.759 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":142,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:50:29.725969+0000","last_up_change":"2024-09-15T02:49:42.289183+0000","last_in_change":"2024-09-15T02:49:52.362323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":74,"up_from":98,"up_thru":106,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":34477},{"type":"v1","addr":"172.21.15.152:6801","nonce":34477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":34477},{"type":"v1","addr":"172.21.15.152:6803","nonce":34477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":34477},{"type":"v1","addr":"172.21.15.152:6807","nonce":34477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":34477},{"type":"v1","addr":"172.21.15.152:6805","nonce":34477}]},"public_addr":"172.21.15.152:6801/34477","cluster_addr":"172.21.15.152:6803/34477","heartbeat_back_addr":"172.21.15.152:6807/34477","heartbeat_front_addr":"172.21.15.152:6805/34477","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":29,"up_from":42,"up_thru":76,"down_at":31,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":27185},{"type":"v1","addr":"172.21.15.152:6812","nonce":27185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":27185},{"type":"v1","addr":"172.21.15.152:6819","nonce":27185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":27185},{"type":"v1","addr":"172.21.15.152:6831","nonce":27185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":27185},{"type":"v1","addr":"172.21.15.152:6826","nonce":27185}]},"public_addr":"172.21.15.152:6812/27185","cluster_addr":"172.21.15.152:6819/27185","heartbeat_back_addr":"172.21.15.152:6831/27185","heartbeat_front_addr":"172.21.15.152:6826/27185","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:47.242372+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:34.269618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:39.185164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.093348+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.890794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:43.303875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:41.805268+0000","dead_epoch":75},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.289669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:36.555828+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:44.408648+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:29.360419+0000","dead_epoch":31},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.224457+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:48.195616+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.117214+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:28.670589+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:50.102496+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:50:34.772 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T02:50:34.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:50:35.074 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:35.078 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:50:35.091 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T02:50:34.989827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4835604,"kb_used_data":26272,"kb_used_omap":36,"kb_used_meta":4809051,"kb_avail":1505113836,"statfs":{"total":1546188226560,"available":1541236568064,"internally_reserved":0,"allocated":26902528,"data_stored":21252846,"data_compressed":3824442,"data_compressed_allocated":7929856,"data_compressed_original":16752640,"omap_allocated":37677,"internal_metadata":4924468435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.270536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.486502+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:59.318029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:59.318029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:26.106992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.955225+0000","last_change":"2024-09-15T02:49:53.389280+0000","last_active":"2024-09-15T02:50:28.955225+0000","last_peered":"2024-09-15T02:50:28.955225+0000","last_clean":"2024-09-15T02:50:28.955225+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:28.955225+0000","last_undegraded":"2024-09-15T02:50:28.955225+0000","last_fullsized":"2024-09-15T02:50:28.955225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.695063+0000","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.154656+0000","last_change":"2024-09-15T02:49:53.385816+0000","last_active":"2024-09-15T02:50:29.154656+0000","last_peered":"2024-09-15T02:50:29.154656+0000","last_clean":"2024-09-15T02:50:29.154656+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:29.154656+0000","last_undegraded":"2024-09-15T02:50:29.154656+0000","last_fullsized":"2024-09-15T02:50:29.154656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:08.412981+0000","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.060075+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:50:29.060075+0000","last_peered":"2024-09-15T02:50:29.060075+0000","last_clean":"2024-09-15T02:50:29.060075+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:29.060075+0000","last_undegraded":"2024-09-15T02:50:29.060075+0000","last_fullsized":"2024-09-15T02:50:29.060075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":64,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953455+0000","last_change":"2024-09-15T02:50:04.489301+0000","last_active":"2024-09-15T02:50:28.953455+0000","last_peered":"2024-09-15T02:50:28.953455+0000","last_clean":"2024-09-15T02:50:28.953455+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:28.953455+0000","last_undegraded":"2024-09-15T02:50:28.953455+0000","last_fullsized":"2024-09-15T02:50:28.953455+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T02:49:56.345917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:56.345917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.722217+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:50:28.722217+0000","last_peered":"2024-09-15T02:50:28.722217+0000","last_clean":"2024-09-15T02:50:28.722217+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:28.722217+0000","last_undegraded":"2024-09-15T02:50:28.722217+0000","last_fullsized":"2024-09-15T02:50:28.722217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.613224+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.853218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.853218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.808351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954684+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:50:28.954684+0000","last_peered":"2024-09-15T02:50:28.954684+0000","last_clean":"2024-09-15T02:50:28.954684+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:28.954684+0000","last_undegraded":"2024-09-15T02:50:28.954684+0000","last_fullsized":"2024-09-15T02:50:28.954684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954732+0000","last_change":"2024-09-15T02:50:03.139015+0000","last_active":"2024-09-15T02:50:28.954732+0000","last_peered":"2024-09-15T02:50:28.954732+0000","last_clean":"2024-09-15T02:50:28.954732+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:28.954732+0000","last_undegraded":"2024-09-15T02:50:28.954732+0000","last_fullsized":"2024-09-15T02:50:28.954732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-15T02:50:03.138800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:50:03.138800+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.267089+0000","scrub_duration":0.0046469529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300144,"kb_used_data":1632,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071696,"statfs":{"total":96636764160,"available":96329416704,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]}]},{"osd":7,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.373}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.498},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.5549999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.246}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.5309999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.73}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.6539999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.754},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.677}]}]},{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.96499999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.2809999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":1.377},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.2450000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.4419999999999999},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":1.321}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.554},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":1.4730000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.6519999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.679},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.5780000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.6100000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.522},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.696}]}]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3759999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.5509999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.7689999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.6830000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.73},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.857}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":50.643000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":2.04}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.68600000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.052},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.0049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.4730000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.73699999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.264},{"interface":"front","average":{"1min":29.276,"5min":29.276,"15min":29.276},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":549.39099999999996,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.413}]},{"osd":12,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":38.726999999999997,"5min":38.726999999999997,"15min":38.726999999999997},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":893.91999999999996,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.579},{"interface":"front","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.6060000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":6.6509999999999998},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":122.18000000000001,"5min":122.18000000000001,"15min":122.18000000000001},"last":1.538},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.2869999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":13.276999999999999,"15min":13.276999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":286.791,"5min":286.791,"15min":286.791},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.3600000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.4570000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":1.0149999999999999},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.728,"15min":1.728},"min":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"min":{"1min":1.512,"5min":1.512,"15min":1.512},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.4399999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"min":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6120000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":27.055,"5min":27.055,"15min":27.055},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":501.60599999999999,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.411},{"interface":"front","average":{"1min":2.157,"5min":2.157,"15min":2.157},"min":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.5089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"min":{"1min":1.466,"5min":1.466,"15min":1.466},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.472},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"min":{"1min":1.7,"5min":1.7,"15min":1.7},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5840000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.357},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.659}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.5580000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.776},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.681}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.365}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.028}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":0.91100000000000003}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.4850000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.3380000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":12.597,"5min":12.597,"15min":12.597},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":189.79599999999999,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":10.946,"5min":10.946,"15min":10.946},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":163.952,"5min":163.952,"15min":163.952},"last":1.129}]},{"osd":13,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.244}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.5189999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.55}]}]},{"osd":14,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":1.24},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":0.92900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":1.095},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.131},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.175}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":0.98999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":1.1639999999999999},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.3720000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.331}]},{"osd":10,"last update":"Sun Sep 15 02:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.756}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":2.0009999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9550000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.7310000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.79500000000000004},{"interface":"front","average":{"1min":30.068000000000001,"5min":30.068000000000001,"15min":30.068000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":611.07899999999995,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.91500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.024}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.258},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.0620000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.179},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.232}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.399},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":1.3240000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.54},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.518}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.29},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.482}]}]},{"osd":15,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":0.72299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.111},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2729999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.3}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3320000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":2.069}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.3839999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.413}]}]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":1.0249999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":1.222},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.329}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.2829999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.2609999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.165},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.3999999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303160,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068680,"statfs":{"total":96636764160,"available":96326328320,"internally_reserved":0,"allocated":1679360,"data_stored":1262348,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":22.859000000000002,"5min":22.859000000000002,"15min":22.859000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":382.21199999999999,"5min":382.21199999999999,"15min":382.21199999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.109},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.3540000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.056},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.486}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":1.579},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.613}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.159},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.6379999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.778},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.2969999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.661},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.71}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.821}]}]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1261770,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":3.8540000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.71499999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.3180000000000001},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":0.88600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.0009999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.0489999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4019999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5069999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.7050000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":4.2119999999999997},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.3640000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.5449999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299504,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072336,"statfs":{"total":96636764160,"available":96330072064,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90600000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":12.095000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":189.78700000000001,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.101},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.165}]},{"osd":4,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":4.0759999999999996,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.409},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.1319999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.238},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2150000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.4410000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.4970000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":9.3740000000000006,"5min":9.3740000000000006,"15min":9.3740000000000006},"last":1.556},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.6220000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":11.145,"5min":11.145,"15min":11.145},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":170.46600000000001,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":17.238,"5min":17.238,"15min":17.238},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":272.40899999999999,"5min":272.40899999999999,"15min":272.40899999999999},"last":1.423}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.585},{"interface":"front","average":{"1min":35.201000000000001,"5min":35.201000000000001,"15min":35.201000000000001},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":603.03999999999996,"5min":603.03999999999996,"15min":603.03999999999996},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.081}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.504},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.5600000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.135},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.625}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.698},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.736}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.796},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.655}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.829},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.7589999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":2.0299999999999998},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.972}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.929}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":2.0760000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.996}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.123}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.874},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.3340000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.431},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.169}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.294},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.202}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.2509999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.5229999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.3320000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.383},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":1.6060000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:50:35.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:50:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:35.416 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:50:35.428 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T02:50:34.989827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4835604,"kb_used_data":26272,"kb_used_omap":36,"kb_used_meta":4809051,"kb_avail":1505113836,"statfs":{"total":1546188226560,"available":1541236568064,"internally_reserved":0,"allocated":26902528,"data_stored":21252846,"data_compressed":3824442,"data_compressed_allocated":7929856,"data_compressed_original":16752640,"omap_allocated":37677,"internal_metadata":4924468435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.270536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.486502+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:59.318029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:59.318029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:26.106992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.955225+0000","last_change":"2024-09-15T02:49:53.389280+0000","last_active":"2024-09-15T02:50:28.955225+0000","last_peered":"2024-09-15T02:50:28.955225+0000","last_clean":"2024-09-15T02:50:28.955225+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:28.955225+0000","last_undegraded":"2024-09-15T02:50:28.955225+0000","last_fullsized":"2024-09-15T02:50:28.955225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.695063+0000","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.154656+0000","last_change":"2024-09-15T02:49:53.385816+0000","last_active":"2024-09-15T02:50:29.154656+0000","last_peered":"2024-09-15T02:50:29.154656+0000","last_clean":"2024-09-15T02:50:29.154656+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:29.154656+0000","last_undegraded":"2024-09-15T02:50:29.154656+0000","last_fullsized":"2024-09-15T02:50:29.154656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:08.412981+0000","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.060075+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:50:29.060075+0000","last_peered":"2024-09-15T02:50:29.060075+0000","last_clean":"2024-09-15T02:50:29.060075+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:29.060075+0000","last_undegraded":"2024-09-15T02:50:29.060075+0000","last_fullsized":"2024-09-15T02:50:29.060075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":64,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953455+0000","last_change":"2024-09-15T02:50:04.489301+0000","last_active":"2024-09-15T02:50:28.953455+0000","last_peered":"2024-09-15T02:50:28.953455+0000","last_clean":"2024-09-15T02:50:28.953455+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:28.953455+0000","last_undegraded":"2024-09-15T02:50:28.953455+0000","last_fullsized":"2024-09-15T02:50:28.953455+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T02:49:56.345917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:56.345917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.722217+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:50:28.722217+0000","last_peered":"2024-09-15T02:50:28.722217+0000","last_clean":"2024-09-15T02:50:28.722217+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:28.722217+0000","last_undegraded":"2024-09-15T02:50:28.722217+0000","last_fullsized":"2024-09-15T02:50:28.722217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.613224+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.853218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.853218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.808351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954684+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:50:28.954684+0000","last_peered":"2024-09-15T02:50:28.954684+0000","last_clean":"2024-09-15T02:50:28.954684+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:28.954684+0000","last_undegraded":"2024-09-15T02:50:28.954684+0000","last_fullsized":"2024-09-15T02:50:28.954684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954732+0000","last_change":"2024-09-15T02:50:03.139015+0000","last_active":"2024-09-15T02:50:28.954732+0000","last_peered":"2024-09-15T02:50:28.954732+0000","last_clean":"2024-09-15T02:50:28.954732+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:28.954732+0000","last_undegraded":"2024-09-15T02:50:28.954732+0000","last_fullsized":"2024-09-15T02:50:28.954732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-15T02:50:03.138800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:50:03.138800+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.267089+0000","scrub_duration":0.0046469529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300144,"kb_used_data":1632,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071696,"statfs":{"total":96636764160,"available":96329416704,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]}]},{"osd":7,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.373}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.498},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.5549999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.246}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.5309999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.73}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.6539999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.754},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.677}]}]},{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.96499999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.2809999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":1.377},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.2450000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.4419999999999999},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":1.321}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.554},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":1.4730000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.6519999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.679},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.5780000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.6100000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.522},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.696}]}]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3759999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.5509999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.7689999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.6830000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.73},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.857}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":50.643000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":2.04}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.68600000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.052},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.0049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.4730000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.73699999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.264},{"interface":"front","average":{"1min":29.276,"5min":29.276,"15min":29.276},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":549.39099999999996,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.413}]},{"osd":12,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":38.726999999999997,"5min":38.726999999999997,"15min":38.726999999999997},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":893.91999999999996,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.579},{"interface":"front","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.6060000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":6.6509999999999998},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":122.18000000000001,"5min":122.18000000000001,"15min":122.18000000000001},"last":1.538},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.2869999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":13.276999999999999,"15min":13.276999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":286.791,"5min":286.791,"15min":286.791},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.3600000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.4570000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":1.0149999999999999},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.728,"15min":1.728},"min":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"min":{"1min":1.512,"5min":1.512,"15min":1.512},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.4399999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"min":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6120000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":27.055,"5min":27.055,"15min":27.055},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":501.60599999999999,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.411},{"interface":"front","average":{"1min":2.157,"5min":2.157,"15min":2.157},"min":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.5089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"min":{"1min":1.466,"5min":1.466,"15min":1.466},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.472},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"min":{"1min":1.7,"5min":1.7,"15min":1.7},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5840000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.357},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.659}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.5580000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.776},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.681}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.365}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.028}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":0.91100000000000003}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.4850000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.3380000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":12.597,"5min":12.597,"15min":12.597},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":189.79599999999999,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":10.946,"5min":10.946,"15min":10.946},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":163.952,"5min":163.952,"15min":163.952},"last":1.129}]},{"osd":13,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.244}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.5189999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.55}]}]},{"osd":14,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":1.24},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":0.92900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":1.095},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.131},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.175}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":0.98999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":1.1639999999999999},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.3720000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.331}]},{"osd":10,"last update":"Sun Sep 15 02:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.756}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":2.0009999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9550000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.7310000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.79500000000000004},{"interface":"front","average":{"1min":30.068000000000001,"5min":30.068000000000001,"15min":30.068000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":611.07899999999995,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.91500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.024}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.258},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.0620000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.179},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.232}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.399},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":1.3240000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.54},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.518}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.29},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.482}]}]},{"osd":15,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":0.72299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.111},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2729999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.3}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3320000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":2.069}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.3839999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.413}]}]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":1.0249999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":1.222},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.329}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.2829999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.2609999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.165},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.3999999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303160,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068680,"statfs":{"total":96636764160,"available":96326328320,"internally_reserved":0,"allocated":1679360,"data_stored":1262348,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":22.859000000000002,"5min":22.859000000000002,"15min":22.859000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":382.21199999999999,"5min":382.21199999999999,"15min":382.21199999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.109},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.3540000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.056},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.486}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":1.579},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.613}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.159},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.6379999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.778},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.2969999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.661},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.71}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.821}]}]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1261770,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":3.8540000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.71499999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.3180000000000001},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":0.88600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.0009999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.0489999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4019999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5069999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.7050000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":4.2119999999999997},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.3640000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.5449999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299504,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072336,"statfs":{"total":96636764160,"available":96330072064,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90600000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":12.095000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":189.78700000000001,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.101},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.165}]},{"osd":4,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":4.0759999999999996,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.409},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.1319999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.238},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2150000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.4410000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.4970000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":9.3740000000000006,"5min":9.3740000000000006,"15min":9.3740000000000006},"last":1.556},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.6220000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":11.145,"5min":11.145,"15min":11.145},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":170.46600000000001,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":17.238,"5min":17.238,"15min":17.238},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":272.40899999999999,"5min":272.40899999999999,"15min":272.40899999999999},"last":1.423}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.585},{"interface":"front","average":{"1min":35.201000000000001,"5min":35.201000000000001,"15min":35.201000000000001},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":603.03999999999996,"5min":603.03999999999996,"15min":603.03999999999996},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.081}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.504},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.5600000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.135},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.625}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.698},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.736}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.796},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.655}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.829},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.7589999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":2.0299999999999998},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.972}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.929}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":2.0760000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.996}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.123}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.874},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.3340000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.431},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.169}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.294},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.202}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.2509999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.5229999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.3320000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.383},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":1.6060000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:50:35.432 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T02:50:35.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:50:35.752 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:50:35.767 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T02:50:34.989827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4835604,"kb_used_data":26272,"kb_used_omap":36,"kb_used_meta":4809051,"kb_avail":1505113836,"statfs":{"total":1546188226560,"available":1541236568064,"internally_reserved":0,"allocated":26902528,"data_stored":21252846,"data_compressed":3824442,"data_compressed_allocated":7929856,"data_compressed_original":16752640,"omap_allocated":37677,"internal_metadata":4924468435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.270536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.486502+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:59.318029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:59.318029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:26.106992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.955225+0000","last_change":"2024-09-15T02:49:53.389280+0000","last_active":"2024-09-15T02:50:28.955225+0000","last_peered":"2024-09-15T02:50:28.955225+0000","last_clean":"2024-09-15T02:50:28.955225+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:28.955225+0000","last_undegraded":"2024-09-15T02:50:28.955225+0000","last_fullsized":"2024-09-15T02:50:28.955225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:00.695063+0000","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.154656+0000","last_change":"2024-09-15T02:49:53.385816+0000","last_active":"2024-09-15T02:50:29.154656+0000","last_peered":"2024-09-15T02:50:29.154656+0000","last_clean":"2024-09-15T02:50:29.154656+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:29.154656+0000","last_undegraded":"2024-09-15T02:50:29.154656+0000","last_fullsized":"2024-09-15T02:50:29.154656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:08.412981+0000","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:29.060075+0000","last_change":"2024-09-15T02:49:29.518085+0000","last_active":"2024-09-15T02:50:29.060075+0000","last_peered":"2024-09-15T02:50:29.060075+0000","last_clean":"2024-09-15T02:50:29.060075+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:29.060075+0000","last_undegraded":"2024-09-15T02:50:29.060075+0000","last_fullsized":"2024-09-15T02:50:29.060075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:29.517881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:29.517881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:50.819881+0000","scrub_duration":0.0034748470000000001,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":64,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953455+0000","last_change":"2024-09-15T02:50:04.489301+0000","last_active":"2024-09-15T02:50:28.953455+0000","last_peered":"2024-09-15T02:50:28.953455+0000","last_clean":"2024-09-15T02:50:28.953455+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:28.953455+0000","last_undegraded":"2024-09-15T02:50:28.953455+0000","last_fullsized":"2024-09-15T02:50:28.953455+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T02:49:56.345917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:56.345917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.722217+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:50:28.722217+0000","last_peered":"2024-09-15T02:50:28.722217+0000","last_clean":"2024-09-15T02:50:28.722217+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:28.722217+0000","last_undegraded":"2024-09-15T02:50:28.722217+0000","last_fullsized":"2024-09-15T02:50:28.722217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.953524+0000","last_change":"2024-09-15T02:50:04.613224+0000","last_active":"2024-09-15T02:50:28.953524+0000","last_peered":"2024-09-15T02:50:28.953524+0000","last_clean":"2024-09-15T02:50:28.953524+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:28.953524+0000","last_undegraded":"2024-09-15T02:50:28.953524+0000","last_fullsized":"2024-09-15T02:50:28.953524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.853218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.853218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.808351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954684+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:50:28.954684+0000","last_peered":"2024-09-15T02:50:28.954684+0000","last_clean":"2024-09-15T02:50:28.954684+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:28.954684+0000","last_undegraded":"2024-09-15T02:50:28.954684+0000","last_fullsized":"2024-09-15T02:50:28.954684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T02:50:28.954732+0000","last_change":"2024-09-15T02:50:03.139015+0000","last_active":"2024-09-15T02:50:28.954732+0000","last_peered":"2024-09-15T02:50:28.954732+0000","last_clean":"2024-09-15T02:50:28.954732+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:28.954732+0000","last_undegraded":"2024-09-15T02:50:28.954732+0000","last_fullsized":"2024-09-15T02:50:28.954732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-15T02:50:03.138800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:50:03.138800+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.267089+0000","scrub_duration":0.0046469529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300144,"kb_used_data":1632,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071696,"statfs":{"total":96636764160,"available":96329416704,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]}]},{"osd":7,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.373}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.498},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.5549999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.246}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.5309999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.73}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.6539999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.754},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.677}]}]},{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.96499999999999997}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.2809999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":1.377},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.2450000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.4419999999999999},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":1.321}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.554},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":1.4730000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.6519999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.679},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.5780000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.6100000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.522},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.696}]}]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3759999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.5509999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.7689999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.6830000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.73},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.857}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":50.643000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":2.04}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1671168,"data_stored":1261751,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.68600000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.052},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":1.0049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.4730000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.73699999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.264},{"interface":"front","average":{"1min":29.276,"5min":29.276,"15min":29.276},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":549.39099999999996,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.413}]},{"osd":12,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":38.726999999999997,"5min":38.726999999999997,"15min":38.726999999999997},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":893.91999999999996,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.579},{"interface":"front","average":{"1min":1.677,"5min":1.677,"15min":1.677},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.6060000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":6.6509999999999998},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":122.18000000000001,"5min":122.18000000000001,"15min":122.18000000000001},"last":1.538},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.2869999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:49:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":13.276999999999999,"15min":13.276999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":286.791,"5min":286.791,"15min":286.791},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.3600000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.4570000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":1.0149999999999999},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.728,"15min":1.728},"min":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"min":{"1min":1.512,"5min":1.512,"15min":1.512},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.4399999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"min":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.6120000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":27.055,"5min":27.055,"15min":27.055},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":501.60599999999999,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.411},{"interface":"front","average":{"1min":2.157,"5min":2.157,"15min":2.157},"min":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.5089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"min":{"1min":1.466,"5min":1.466,"15min":1.466},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.472},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"min":{"1min":1.7,"5min":1.7,"15min":1.7},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5840000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.357},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.659}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.5580000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.776},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.681}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.365}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.028}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":0.91100000000000003}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.4850000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.3380000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":12.597,"5min":12.597,"15min":12.597},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":189.79599999999999,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":10.946,"5min":10.946,"15min":10.946},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":163.952,"5min":163.952,"15min":163.952},"last":1.129}]},{"osd":13,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.244}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.5189999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.55}]}]},{"osd":14,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":1.24},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":0.92900000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":1.095},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.131},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.175}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":0.98999999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":1.1639999999999999},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.3720000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.331}]},{"osd":10,"last update":"Sun Sep 15 02:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.756}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":2.0009999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9550000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.7310000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.79500000000000004},{"interface":"front","average":{"1min":30.068000000000001,"5min":30.068000000000001,"15min":30.068000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":611.07899999999995,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.91500000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.024}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.258},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.0620000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.179},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.232}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.399},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":1.3240000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.54},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.518}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.29},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.482}]}]},{"osd":15,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":0.72299999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.111},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2729999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.3}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3320000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":2.069}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.3839999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.413}]}]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303156,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":1.0249999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":1.222},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.329}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.2829999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.2609999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.165},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.3999999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303160,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068680,"statfs":{"total":96636764160,"available":96326328320,"internally_reserved":0,"allocated":1679360,"data_stored":1262348,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":22.859000000000002,"5min":22.859000000000002,"15min":22.859000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":382.21199999999999,"5min":382.21199999999999,"15min":382.21199999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.109},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.3540000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.056},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.486}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":1.579},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.613}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.159},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.6379999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.778},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.2969999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.661},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.71}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.821}]}]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1261770,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":3.8540000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.71499999999999997}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.3180000000000001},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":0.88600000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.0009999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.0489999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.4019999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5069999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.7050000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":4.2119999999999997},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.3640000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.5449999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299504,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072336,"statfs":{"total":96636764160,"available":96330072064,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90600000000000003}]},{"osd":1,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":12.095000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":189.78700000000001,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.101},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.165}]},{"osd":4,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":4.0759999999999996,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.409},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.1319999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.238},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2150000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.4410000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.4970000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":9.3740000000000006,"5min":9.3740000000000006,"15min":9.3740000000000006},"last":1.556},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.6220000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":11.145,"5min":11.145,"15min":11.145},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":170.46600000000001,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":17.238,"5min":17.238,"15min":17.238},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":272.40899999999999,"5min":272.40899999999999,"15min":272.40899999999999},"last":1.423}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.585},{"interface":"front","average":{"1min":35.201000000000001,"5min":35.201000000000001,"15min":35.201000000000001},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":603.03999999999996,"5min":603.03999999999996,"15min":603.03999999999996},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1791257,"data_compressed":245682,"data_compressed_allocated":524288,"data_compressed_original":1495040,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.081}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.504},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.5600000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.135},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.625}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.698},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.736}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.796},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.655}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.829},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.7589999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":2.0299999999999998},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.972}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.929}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":2.0760000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.996}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298996,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072844,"statfs":{"total":96636764160,"available":96330592256,"internally_reserved":0,"allocated":1675264,"data_stored":1262329,"data_compressed":238077,"data_compressed_allocated":491520,"data_compressed_original":983040,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.123}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.874},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.96199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.3340000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.431},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.169}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.294},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.202}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.2509999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.5229999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.3320000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.383},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":1.6060000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:50:35.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T02:50:36.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:36.073 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":144,"fsid":"24e12693-a4e4-4f9a-bb1d-999f41be75a2","created":"2024-09-15T02:48:10.478850+0000","modified":"2024-09-15T02:50:35.771007+0000","last_up_change":"2024-09-15T02:49:42.289183+0000","last_in_change":"2024-09-15T02:49:52.362323+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T02:48:23.027168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T02:48:25.515637+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2d28679c-afcb-472a-8a92-b30cbeb553a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":22132},{"type":"v1","addr":"172.21.15.6:6817","nonce":22132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":22132},{"type":"v1","addr":"172.21.15.6:6819","nonce":22132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":22132},{"type":"v1","addr":"172.21.15.6:6823","nonce":22132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":22132},{"type":"v1","addr":"172.21.15.6:6821","nonce":22132}]},"public_addr":"172.21.15.6:6817/22132","cluster_addr":"172.21.15.6:6819/22132","heartbeat_back_addr":"172.21.15.6:6823/22132","heartbeat_front_addr":"172.21.15.6:6821/22132","state":["exists","up"]},{"osd":1,"uuid":"a3111517-1d9f-4c6b-a838-8eb5e07f8c39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":22077},{"type":"v1","addr":"172.21.15.118:6810","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":22077},{"type":"v1","addr":"172.21.15.118:6814","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":22077},{"type":"v1","addr":"172.21.15.118:6822","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":22077},{"type":"v1","addr":"172.21.15.118:6818","nonce":22077}]},"public_addr":"172.21.15.118:6810/22077","cluster_addr":"172.21.15.118:6814/22077","heartbeat_back_addr":"172.21.15.118:6822/22077","heartbeat_front_addr":"172.21.15.118:6818/22077","state":["exists","up"]},{"osd":2,"uuid":"5a893db6-a9ac-4c69-ac36-57826eab4e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":22074},{"type":"v1","addr":"172.21.15.152:6813","nonce":22074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":22074},{"type":"v1","addr":"172.21.15.152:6818","nonce":22074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6825","nonce":22074},{"type":"v1","addr":"172.21.15.152:6827","nonce":22074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":22074},{"type":"v1","addr":"172.21.15.152:6824","nonce":22074}]},"public_addr":"172.21.15.152:6813/22074","cluster_addr":"172.21.15.152:6818/22074","heartbeat_back_addr":"172.21.15.152:6827/22074","heartbeat_front_addr":"172.21.15.152:6824/22074","state":["exists","up"]},{"osd":3,"uuid":"ab0d2e8a-aedd-4d8b-a5d4-881317345d27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6823","nonce":22466},{"type":"v1","addr":"172.21.15.191:6827","nonce":22466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":22466},{"type":"v1","addr":"172.21.15.191:6829","nonce":22466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":22466},{"type":"v1","addr":"172.21.15.191:6833","nonce":22466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":22466},{"type":"v1","addr":"172.21.15.191:6831","nonce":22466}]},"public_addr":"172.21.15.191:6827/22466","cluster_addr":"172.21.15.191:6829/22466","heartbeat_back_addr":"172.21.15.191:6833/22466","heartbeat_front_addr":"172.21.15.191:6831/22466","state":["exists","up"]},{"osd":4,"uuid":"3edf8310-01a7-4377-a469-b376971149ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":22131},{"type":"v1","addr":"172.21.15.6:6801","nonce":22131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":22131},{"type":"v1","addr":"172.21.15.6:6803","nonce":22131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":22131},{"type":"v1","addr":"172.21.15.6:6807","nonce":22131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":22131},{"type":"v1","addr":"172.21.15.6:6805","nonce":22131}]},"public_addr":"172.21.15.6:6801/22131","cluster_addr":"172.21.15.6:6803/22131","heartbeat_back_addr":"172.21.15.6:6807/22131","heartbeat_front_addr":"172.21.15.6:6805/22131","state":["exists","up"]},{"osd":5,"uuid":"8000b2e4-ca30-4591-88f2-9b0eee27cbca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":22078},{"type":"v1","addr":"172.21.15.118:6801","nonce":22078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":22078},{"type":"v1","addr":"172.21.15.118:6803","nonce":22078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":22078},{"type":"v1","addr":"172.21.15.118:6807","nonce":22078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":22078},{"type":"v1","addr":"172.21.15.118:6805","nonce":22078}]},"public_addr":"172.21.15.118:6801/22078","cluster_addr":"172.21.15.118:6803/22078","heartbeat_back_addr":"172.21.15.118:6807/22078","heartbeat_front_addr":"172.21.15.118:6805/22078","state":["exists","up"]},{"osd":6,"uuid":"7f8ecd2a-7829-496b-b349-5fdebbb6962a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":74,"up_from":98,"up_thru":106,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":34477},{"type":"v1","addr":"172.21.15.152:6801","nonce":34477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":34477},{"type":"v1","addr":"172.21.15.152:6803","nonce":34477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":34477},{"type":"v1","addr":"172.21.15.152:6807","nonce":34477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":34477},{"type":"v1","addr":"172.21.15.152:6805","nonce":34477}]},"public_addr":"172.21.15.152:6801/34477","cluster_addr":"172.21.15.152:6803/34477","heartbeat_back_addr":"172.21.15.152:6807/34477","heartbeat_front_addr":"172.21.15.152:6805/34477","state":["exists","up"]},{"osd":7,"uuid":"d94d0835-cb78-46f5-ae93-1b6d60a9ad84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":22456},{"type":"v1","addr":"172.21.15.191:6803","nonce":22456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":22456},{"type":"v1","addr":"172.21.15.191:6805","nonce":22456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":22456},{"type":"v1","addr":"172.21.15.191:6809","nonce":22456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":22456},{"type":"v1","addr":"172.21.15.191:6807","nonce":22456}]},"public_addr":"172.21.15.191:6803/22456","cluster_addr":"172.21.15.191:6805/22456","heartbeat_back_addr":"172.21.15.191:6809/22456","heartbeat_front_addr":"172.21.15.191:6807/22456","state":["exists","up"]},{"osd":8,"uuid":"61e692a5-d458-4bbc-8678-66f2c7b6bfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":22164},{"type":"v1","addr":"172.21.15.6:6809","nonce":22164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":22164},{"type":"v1","addr":"172.21.15.6:6811","nonce":22164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":22164},{"type":"v1","addr":"172.21.15.6:6815","nonce":22164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":22164},{"type":"v1","addr":"172.21.15.6:6813","nonce":22164}]},"public_addr":"172.21.15.6:6809/22164","cluster_addr":"172.21.15.6:6811/22164","heartbeat_back_addr":"172.21.15.6:6815/22164","heartbeat_front_addr":"172.21.15.6:6813/22164","state":["exists","up"]},{"osd":9,"uuid":"b9a7b347-f334-4ccc-82ad-d6d67b366485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":22079},{"type":"v1","addr":"172.21.15.118:6811","nonce":22079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":22079},{"type":"v1","addr":"172.21.15.118:6815","nonce":22079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":22079},{"type":"v1","addr":"172.21.15.118:6823","nonce":22079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":22079},{"type":"v1","addr":"172.21.15.118:6819","nonce":22079}]},"public_addr":"172.21.15.118:6811/22079","cluster_addr":"172.21.15.118:6815/22079","heartbeat_back_addr":"172.21.15.118:6823/22079","heartbeat_front_addr":"172.21.15.118:6819/22079","state":["exists","up"]},{"osd":10,"uuid":"f446f37d-a5c3-4e59-88f9-5a0bb818b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":29,"up_from":42,"up_thru":76,"down_at":31,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":27185},{"type":"v1","addr":"172.21.15.152:6812","nonce":27185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":27185},{"type":"v1","addr":"172.21.15.152:6819","nonce":27185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6829","nonce":27185},{"type":"v1","addr":"172.21.15.152:6831","nonce":27185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":27185},{"type":"v1","addr":"172.21.15.152:6826","nonce":27185}]},"public_addr":"172.21.15.152:6812/27185","cluster_addr":"172.21.15.152:6819/27185","heartbeat_back_addr":"172.21.15.152:6831/27185","heartbeat_front_addr":"172.21.15.152:6826/27185","state":["exists","up"]},{"osd":11,"uuid":"78916195-e38b-4ae2-a160-6feb89ec76ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":22467},{"type":"v1","addr":"172.21.15.191:6819","nonce":22467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":22467},{"type":"v1","addr":"172.21.15.191:6821","nonce":22467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":22467},{"type":"v1","addr":"172.21.15.191:6826","nonce":22467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":22467},{"type":"v1","addr":"172.21.15.191:6824","nonce":22467}]},"public_addr":"172.21.15.191:6819/22467","cluster_addr":"172.21.15.191:6821/22467","heartbeat_back_addr":"172.21.15.191:6826/22467","heartbeat_front_addr":"172.21.15.191:6824/22467","state":["exists","up"]},{"osd":12,"uuid":"778d9a09-0481-4af0-b838-ceeabc6c0af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":22165},{"type":"v1","addr":"172.21.15.6:6825","nonce":22165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":22165},{"type":"v1","addr":"172.21.15.6:6827","nonce":22165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":22165},{"type":"v1","addr":"172.21.15.6:6831","nonce":22165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":22165},{"type":"v1","addr":"172.21.15.6:6829","nonce":22165}]},"public_addr":"172.21.15.6:6825/22165","cluster_addr":"172.21.15.6:6827/22165","heartbeat_back_addr":"172.21.15.6:6831/22165","heartbeat_front_addr":"172.21.15.6:6829/22165","state":["exists","up"]},{"osd":13,"uuid":"5167c2e2-7879-4a97-9966-f0621d9ced72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":22080},{"type":"v1","addr":"172.21.15.118:6825","nonce":22080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":22080},{"type":"v1","addr":"172.21.15.118:6827","nonce":22080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":22080},{"type":"v1","addr":"172.21.15.118:6831","nonce":22080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":22080},{"type":"v1","addr":"172.21.15.118:6829","nonce":22080}]},"public_addr":"172.21.15.118:6825/22080","cluster_addr":"172.21.15.118:6827/22080","heartbeat_back_addr":"172.21.15.118:6831/22080","heartbeat_front_addr":"172.21.15.118:6829/22080","state":["exists","up"]},{"osd":14,"uuid":"7b4d6903-aa33-4280-89f1-05c499290d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":22077},{"type":"v1","addr":"172.21.15.152:6811","nonce":22077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":22077},{"type":"v1","addr":"172.21.15.152:6817","nonce":22077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":22077},{"type":"v1","addr":"172.21.15.152:6830","nonce":22077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":22077},{"type":"v1","addr":"172.21.15.152:6823","nonce":22077}]},"public_addr":"172.21.15.152:6811/22077","cluster_addr":"172.21.15.152:6817/22077","heartbeat_back_addr":"172.21.15.152:6830/22077","heartbeat_front_addr":"172.21.15.152:6823/22077","state":["exists","up"]},{"osd":15,"uuid":"baf2647e-ce90-42a7-82ae-ec7578ef31da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":22457},{"type":"v1","addr":"172.21.15.191:6811","nonce":22457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":22457},{"type":"v1","addr":"172.21.15.191:6813","nonce":22457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":22457},{"type":"v1","addr":"172.21.15.191:6817","nonce":22457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":22457},{"type":"v1","addr":"172.21.15.191:6815","nonce":22457}]},"public_addr":"172.21.15.191:6811/22457","cluster_addr":"172.21.15.191:6813/22457","heartbeat_back_addr":"172.21.15.191:6817/22457","heartbeat_front_addr":"172.21.15.191:6815/22457","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:47.242372+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:34.269618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:39.185164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.093348+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:50:35.353003+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:43.303875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:41.805268+0000","dead_epoch":75},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.289669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:36.555828+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:44.408648+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:50:34.246875+0000","dead_epoch":31},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:32.224457+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:48.195616+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:38.117214+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:28.670589+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T02:49:50.102496+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T02:50:37.089 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T02:50:37.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:37.263 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:37.263 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:37.264 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:37.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T02:50:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T02:50:37.597 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T02:50:37.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:37.758 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:37.758 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:37.758 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:37.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T02:50:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T02:50:38.279 INFO:tasks.ceph:Scrubbing osd.8 2024-09-15T02:50:38.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:38.441 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:38.441 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:38.441 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:38.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-15T02:50:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 8 to deep-scrub 2024-09-15T02:50:38.789 INFO:tasks.ceph:Scrubbing osd.12 2024-09-15T02:50:38.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:38.954 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:38.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-15T02:50:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 12 to deep-scrub 2024-09-15T02:50:39.292 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T02:50:39.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:39.454 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:39.454 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:39.454 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:39.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T02:50:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T02:50:39.802 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T02:50:39.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:39.955 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:39.955 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:39.955 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:39.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T02:50:40.284 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T02:50:40.295 INFO:tasks.ceph:Scrubbing osd.9 2024-09-15T02:50:40.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:40.447 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:40.447 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:40.447 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:40.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-15T02:50:40.775 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 9 to deep-scrub 2024-09-15T02:50:40.784 INFO:tasks.ceph:Scrubbing osd.13 2024-09-15T02:50:40.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:40.942 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:40.942 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:40.942 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:40.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-15T02:50:41.265 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 13 to deep-scrub 2024-09-15T02:50:41.276 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T02:50:41.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:41.434 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:41.434 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:41.434 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:41.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T02:50:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T02:50:41.774 INFO:tasks.ceph:Scrubbing osd.6 2024-09-15T02:50:41.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:41.937 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:41.937 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:41.937 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:41.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-15T02:50:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 6 to deep-scrub 2024-09-15T02:50:42.265 INFO:tasks.ceph:Scrubbing osd.10 2024-09-15T02:50:42.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:42.430 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:42.430 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:42.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-15T02:50:42.769 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 10 to deep-scrub 2024-09-15T02:50:42.779 INFO:tasks.ceph:Scrubbing osd.14 2024-09-15T02:50:42.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:42.942 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:42.942 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:42.942 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:42.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-15T02:50:43.274 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 14 to deep-scrub 2024-09-15T02:50:43.285 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T02:50:43.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:43.446 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:43.446 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:43.446 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:43.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T02:50:43.769 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T02:50:43.785 INFO:tasks.ceph:Scrubbing osd.7 2024-09-15T02:50:43.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:43.945 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:43.945 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:43.945 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:43.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-15T02:50:44.269 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 7 to deep-scrub 2024-09-15T02:50:44.280 INFO:tasks.ceph:Scrubbing osd.11 2024-09-15T02:50:44.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:44.438 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:44.438 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:44.438 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:44.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-15T02:50:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 11 to deep-scrub 2024-09-15T02:50:44.769 INFO:tasks.ceph:Scrubbing osd.15 2024-09-15T02:50:44.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-15T02:50:44.925 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-15T02:50:44.925 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T02:50:44.925 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-15T02:50:44.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-15T02:50:45.245 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 15 to deep-scrub 2024-09-15T02:50:45.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:50:45.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:50:45.568 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:50:45.581 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T02:50:44.993206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4840460,"kb_used_data":27004,"kb_used_omap":36,"kb_used_meta":4813211,"kb_avail":1505108980,"statfs":{"total":1546188226560,"available":1541231595520,"internally_reserved":0,"allocated":27652096,"data_stored":21833322,"data_compressed":3946426,"data_compressed_allocated":8179712,"data_compressed_original":17252352,"omap_allocated":37677,"internal_metadata":4928728275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002025"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":77,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T02:50:42.198118+0000","last_change":"2024-09-15T02:50:42.198027+0000","last_active":"2024-09-15T02:50:42.198118+0000","last_peered":"2024-09-15T02:50:42.198118+0000","last_clean":"2024-09-15T02:50:42.198118+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:42.198118+0000","last_undegraded":"2024-09-15T02:50:42.198118+0000","last_fullsized":"2024-09-15T02:50:42.198118+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:42.197706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:42.197706+0000","last_clean_scrub_stamp":"2024-09-15T02:50:42.197706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:02.767344+0000","scrub_duration":0.003710718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T02:50:41.178989+0000","last_change":"2024-09-15T02:49:53.389280+0000","last_active":"2024-09-15T02:50:41.178989+0000","last_peered":"2024-09-15T02:50:41.178989+0000","last_clean":"2024-09-15T02:50:41.178989+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:41.178989+0000","last_undegraded":"2024-09-15T02:50:41.178989+0000","last_fullsized":"2024-09-15T02:50:41.178989+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:43.343135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:43.343135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0035447819999999998,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":43,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T02:50:41.415630+0000","last_change":"2024-09-15T02:49:53.385816+0000","last_active":"2024-09-15T02:50:41.415630+0000","last_peered":"2024-09-15T02:50:41.415630+0000","last_clean":"2024-09-15T02:50:41.415630+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:41.415630+0000","last_undegraded":"2024-09-15T02:50:41.415630+0000","last_fullsized":"2024-09-15T02:50:41.415630+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:42.472966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:42.472966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0028802659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":145,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T02:50:40.664850+0000","last_change":"2024-09-15T02:50:40.476884+0000","last_active":"2024-09-15T02:50:40.664850+0000","last_peered":"2024-09-15T02:50:40.664850+0000","last_clean":"2024-09-15T02:50:40.664850+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:40.664850+0000","last_undegraded":"2024-09-15T02:50:40.664850+0000","last_fullsized":"2024-09-15T02:50:40.664850+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:40.476570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:40.476570+0000","last_clean_scrub_stamp":"2024-09-15T02:50:40.476570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:04.142168+0000","scrub_duration":0.011617722,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"146'3","reported_seq":80,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T02:50:41.206575+0000","last_change":"2024-09-15T02:50:41.206526+0000","last_active":"2024-09-15T02:50:41.206575+0000","last_peered":"2024-09-15T02:50:41.206575+0000","last_clean":"2024-09-15T02:50:41.206575+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:41.206575+0000","last_undegraded":"2024-09-15T02:50:41.206575+0000","last_fullsized":"2024-09-15T02:50:41.206575+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'3","last_scrub_stamp":"2024-09-15T02:50:41.206367+0000","last_deep_scrub":"146'3","last_deep_scrub_stamp":"2024-09-15T02:50:41.206367+0000","last_clean_scrub_stamp":"2024-09-15T02:50:41.206367+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:03.278836+0000","scrub_duration":0.0060310169999999996,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-15T02:50:36.767719+0000","last_change":"2024-09-15T02:49:44.949381+0000","last_active":"2024-09-15T02:50:36.767719+0000","last_peered":"2024-09-15T02:50:36.767719+0000","last_clean":"2024-09-15T02:50:36.767719+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:36.767719+0000","last_undegraded":"2024-09-15T02:50:36.767719+0000","last_fullsized":"2024-09-15T02:50:36.767719+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:49:44.949059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:44.949059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:58.740630+0000","scrub_duration":0.0027115020000000002,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T02:50:41.080476+0000","last_change":"2024-09-15T02:50:39.190558+0000","last_active":"2024-09-15T02:50:41.080476+0000","last_peered":"2024-09-15T02:50:41.080476+0000","last_clean":"2024-09-15T02:50:41.080476+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:41.080476+0000","last_undegraded":"2024-09-15T02:50:41.080476+0000","last_fullsized":"2024-09-15T02:50:41.080476+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:39.190360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:39.190360+0000","last_clean_scrub_stamp":"2024-09-15T02:50:39.190360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:41.335481+0000","scrub_duration":0.0036948229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":142,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T02:50:39.803004+0000","last_change":"2024-09-15T02:49:46.014909+0000","last_active":"2024-09-15T02:50:39.803004+0000","last_peered":"2024-09-15T02:50:39.803004+0000","last_clean":"2024-09-15T02:50:39.803004+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:39.803004+0000","last_undegraded":"2024-09-15T02:50:39.803004+0000","last_fullsized":"2024-09-15T02:50:39.803004+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T02:49:46.014633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:26.377691+0000","last_clean_scrub_stamp":"2024-09-15T02:49:46.014633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:06.149106+0000","scrub_duration":0.0085046140000000006,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":275,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T02:50:39.803013+0000","last_change":"2024-09-15T02:50:03.139015+0000","last_active":"2024-09-15T02:50:39.803013+0000","last_peered":"2024-09-15T02:50:39.803013+0000","last_clean":"2024-09-15T02:50:39.803013+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:39.803013+0000","last_undegraded":"2024-09-15T02:50:39.803013+0000","last_fullsized":"2024-09-15T02:50:39.803013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-15T02:50:03.138800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:48:23.322558+0000","last_clean_scrub_stamp":"2024-09-15T02:50:03.138800+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:09.267089+0000","scrub_duration":0.0046469529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795022,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300188,"kb_used_data":1680,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071652,"statfs":{"total":96636764160,"available":96329371648,"internally_reserved":0,"allocated":1720320,"data_stored":1299815,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]}]},{"osd":7,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":11.964,"5min":11.964,"15min":11.964},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.98699999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.3169999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.022}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.427}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.47},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.1930000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.042},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.2709999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.365},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.669}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.236},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.5}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.623}]}]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.61799999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.121}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":607.61900000000003},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.51100000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.044},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":0.72399999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":0.76100000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:49:43 2024","interfaces":[{"interface":"back","average":{"1min":39.683,"5min":39.683,"15min":39.683},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":807.56899999999996,"5min":807.56899999999996,"15min":807.56899999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.53200000000000003}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.67900000000000005}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":0.78500000000000003}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.73499999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303176,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068664,"statfs":{"total":96636764160,"available":96326311936,"internally_reserved":0,"allocated":1695744,"data_stored":1280783,"data_compressed":242076,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.95099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.097},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.125}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":0.996},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3340000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.3009999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.278},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.4239999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.228}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.262},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.397}]},{"osd":14,"last update":"Sun Sep 15 02:49:44 2024","interfaces":[{"interface":"back","average":{"1min":48.771999999999998,"5min":48.771999999999998,"15min":48.771999999999998},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":899.98000000000002,"5min":899.98000000000002,"15min":899.98000000000002},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":1.6890000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303200,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1720320,"data_stored":1299815,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.68000000000000005}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":0.747},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":0.995},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.0369999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.004}]},{"osd":11,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.397},"min":{"1min":0.91000000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.651,"5min":23.751000000000001,"15min":27.434000000000001},"min":{"1min":1.133,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.75,"5min":549.39099999999996,"15min":549.39099999999996},"last":0.97499999999999998}]},{"osd":12,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":31.309000000000001,"15min":36.253999999999998},"min":{"1min":0.90200000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.508,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.762,"5min":1.694,"15min":1.6819999999999999},"min":{"1min":1.1679999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.2070000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":5.6760000000000002,"15min":6.3259999999999996},"min":{"1min":1.1100000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8079999999999998,"5min":122.18000000000001,"15min":122.18000000000001},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.3320000000000001,"15min":1.3180000000000001},"min":{"1min":0.83899999999999997,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.0569999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":6.657,"5min":11.952999999999999,"15min":12.835000000000001},"min":{"1min":0.94299999999999995,"5min":0.747,"15min":0.747},"max":{"1min":85.116,"5min":286.791,"15min":286.791},"last":1.157},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.478,"15min":1.4810000000000001},"min":{"1min":1.0049999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.2000000000000002,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.087}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.135}]}]},{"osd":13,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":0.77900000000000003},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":0.85599999999999998}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.70899999999999996}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.643,"15min":1.6990000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.994,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.7589999999999999,"15min":1.7969999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.5699999999999998,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":1.0760000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.171},{"interface":"front","average":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"min":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.4390000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":27.055,"5min":27.055,"15min":27.055},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":501.60599999999999,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.198},{"interface":"front","average":{"1min":2.157,"5min":2.157,"15min":2.157},"min":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.337}]},{"osd":11,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.8420000000000001,"15min":1.8640000000000001},"min":{"1min":1.111,"5min":1.111,"15min":1.111},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":1.111},{"interface":"front","average":{"1min":1.746,"5min":2.0179999999999998,"15min":2.0640000000000001},"min":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"max":{"1min":2.4180000000000001,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.28}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.1479999999999999},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.3759999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.2350000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.256}]}]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":0.85699999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.056}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.135},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.105}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":0.98599999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":1.238},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.4219999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.72},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.782}]},{"osd":11,"last update":"Sun Sep 15 02:50:40 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":10.401,"15min":11.865},"min":{"1min":1.1930000000000001,"5min":1.038,"15min":1.038},"max":{"1min":2.0259999999999998,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.284,"5min":9.0129999999999999,"15min":10.301},"min":{"1min":0.878,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.8,"5min":163.952,"15min":163.952},"last":1.7569999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.538,"15min":1.548},"min":{"1min":1.137,"5min":1.052,"15min":1.052},"max":{"1min":1.95,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.4550000000000001,"15min":1.4750000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6859999999999999,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.663}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.855},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.8340000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:49:39 2024","interfaces":[{"interface":"back","average":{"1min":29.907,"5min":29.907,"15min":29.907},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":510.72500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.8,"5min":1.8,"15min":1.8},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":6.5529999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.4430000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":1.0669999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.752}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.874},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":0.96399999999999997}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.131},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":1.014}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":0.94499999999999995},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.1619999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.048},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.2110000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.101},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.1879999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":1.3180000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.2829999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.3400000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1829321,"data_compressed":253681,"data_compressed_allocated":540672,"data_compressed_original":1527808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.93799999999999994,"15min":0.92600000000000005},"min":{"1min":0.623,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.623},{"interface":"front","average":{"1min":1.091,"5min":24.271999999999998,"15min":28.135999999999999},"min":{"1min":0.76400000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.762,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.76400000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":2.1179999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.2050000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":2.0939999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.80900000000000005}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.101},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":2.214}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0489999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.2749999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.038},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":1.3180000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.254},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.357}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":2.2389999999999999},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.0580000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":1.248}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.206}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.744},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.355}]},{"osd":8,"last update":"Sun Sep 15 02:49:46 2024","interfaces":[{"interface":"back","average":{"1min":22.667000000000002,"5min":22.667000000000002,"15min":22.667000000000002},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":488.12900000000002,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.6240000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":192.24299999999999},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.657}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.581},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.901}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.869},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.829}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.962},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.9319999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":1.012}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":1.3899999999999999},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1.21}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.081},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.246}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":187.041}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.74},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.3500000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.6140000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":1.4970000000000001},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.8100000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.671},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.5589999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.8360000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280543,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303208,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068632,"statfs":{"total":96636764160,"available":96326279168,"internally_reserved":0,"allocated":1728512,"data_stored":1300412,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.6950000000000001,"15min":1.823},"min":{"1min":0.64100000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.534,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":63.902000000000001,"5min":31.067,"15min":25.594999999999999},"min":{"1min":0.61499999999999999,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":979.23099999999999,"5min":979.23099999999999,"15min":979.23099999999999},"last":0.97499999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.712999999999999,"15min":13.712999999999999},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":276.096,"5min":276.096,"15min":276.096},"last":1.1819999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.014,"15min":1.018},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.626},{"interface":"front","average":{"1min":1.226,"5min":1.1200000000000001,"15min":1.1020000000000001},"min":{"1min":0.77200000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":0.77200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.347},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.1140000000000001}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":0.88400000000000001},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.0029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.196},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.135}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.044},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.278}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.173},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.236}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.024}]}]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303200,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068640,"statfs":{"total":96636764160,"available":96326287360,"internally_reserved":0,"allocated":1724416,"data_stored":1299834,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.79600000000000004}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.032},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":1.5469999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.1339999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.446}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:35 2024","interfaces":[{"interface":"back","average":{"1min":24.364999999999998,"5min":5.8689999999999998,"15min":2.786},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":373.05500000000001,"5min":373.05500000000001,"15min":373.05500000000001},"last":1.24},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1819999999999999,"15min":1.196},"min":{"1min":0.745,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8240000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.575}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":1.284},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.4139999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.486},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.3340000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.6419999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.7050000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626458,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299540,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072300,"statfs":{"total":96636764160,"available":96330035200,"internally_reserved":0,"allocated":1712128,"data_stored":1290877,"data_compressed":244076,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.84999999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.3160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:49:52 2024","interfaces":[{"interface":"back","average":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":12.095000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":189.78700000000001,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.077},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.21}]},{"osd":4,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":4.0759999999999996,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.044}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.1020000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.153},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.556}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.706}]},{"osd":9,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":9.3740000000000006,"5min":9.3740000000000006,"15min":9.3740000000000006},"last":1.264},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.663}]},{"osd":11,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":11.145,"5min":11.145,"15min":11.145},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":170.46600000000001,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.24},{"interface":"front","average":{"1min":17.238,"5min":17.238,"15min":17.238},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":272.40899999999999,"5min":272.40899999999999,"15min":272.40899999999999},"last":1.4630000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.746},{"interface":"front","average":{"1min":35.201000000000001,"5min":35.201000000000001,"15min":35.201000000000001},"min":{"1min":1.042,"5min":1.042,"15min":1.042},"max":{"1min":603.03999999999996,"5min":603.03999999999996,"15min":603.03999999999996},"last":666.36099999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303256,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068584,"statfs":{"total":96636764160,"available":96326230016,"internally_reserved":0,"allocated":1777664,"data_stored":1829321,"data_compressed":253681,"data_compressed_allocated":540672,"data_compressed_original":1527808,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.036}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.292}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.1679999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.288},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.3089999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:49:47 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.492},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":1.1910000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.3240000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":1.7},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.6759999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.6559999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.526}]}]},{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299040,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072800,"statfs":{"total":96636764160,"available":96330547200,"internally_reserved":0,"allocated":1724416,"data_stored":1300393,"data_compressed":246076,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.1719999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.345},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.0609999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.034,"5min":1.034,"15min":1.034},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.375}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":0.753}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.149},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.83599999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.4019999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.198}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.008}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.272},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":1.427}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:50:45.587 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T02:49:43.343135+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=49, tm_sec=43, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=50, tm_sec=36, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:50:45.587 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T02:49:42.472966+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=49, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=50, tm_sec=36, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:50:45.588 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T02:49:44.949059+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=49, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=50, tm_sec=36, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:50:45.588 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T02:49:46.014633+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=49, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=50, tm_sec=36, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:50:45.588 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T02:50:03.138800+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=50, tm_sec=3, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=2, tm_min=50, tm_sec=36, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T02:50:45.588 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T02:51:05.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T02:51:05.922 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-15T02:51:05.924 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-15T02:51:05.937 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-15T02:51:04.999167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4841272,"kb_used_data":27808,"kb_used_omap":36,"kb_used_meta":4813211,"kb_avail":1505108168,"statfs":{"total":1546188226560,"available":1541230764032,"internally_reserved":0,"allocated":28475392,"data_stored":22471694,"data_compressed":4080458,"data_compressed_allocated":8454144,"data_compressed_original":17801216,"omap_allocated":37677,"internal_metadata":4928728275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.924880"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":80,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T02:50:47.481367+0000","last_change":"2024-09-15T02:50:42.198027+0000","last_active":"2024-09-15T02:50:47.481367+0000","last_peered":"2024-09-15T02:50:47.481367+0000","last_clean":"2024-09-15T02:50:47.481367+0000","last_became_active":"2024-09-15T02:50:04.485735+0000","last_became_peered":"2024-09-15T02:50:04.485735+0000","last_unstale":"2024-09-15T02:50:47.481367+0000","last_undegraded":"2024-09-15T02:50:47.481367+0000","last_fullsized":"2024-09-15T02:50:47.481367+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:42.197706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:42.197706+0000","last_clean_scrub_stamp":"2024-09-15T02:50:42.197706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:02.767344+0000","scrub_duration":0.003710718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":171,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T02:50:48.236005+0000","last_change":"2024-09-15T02:50:48.235903+0000","last_active":"2024-09-15T02:50:48.236005+0000","last_peered":"2024-09-15T02:50:48.236005+0000","last_clean":"2024-09-15T02:50:48.236005+0000","last_became_active":"2024-09-15T02:49:53.388897+0000","last_became_peered":"2024-09-15T02:49:53.388897+0000","last_unstale":"2024-09-15T02:50:48.236005+0000","last_undegraded":"2024-09-15T02:50:48.236005+0000","last_fullsized":"2024-09-15T02:50:48.236005+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:48.235714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:48.235714+0000","last_clean_scrub_stamp":"2024-09-15T02:50:48.235714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:56.324429+0000","scrub_duration":0.0047044490000000003,"objects_trimmed":0,"snaptrim_duration":0.00090037299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T02:50:47.295512+0000","last_change":"2024-09-15T02:50:43.794103+0000","last_active":"2024-09-15T02:50:47.295512+0000","last_peered":"2024-09-15T02:50:47.295512+0000","last_clean":"2024-09-15T02:50:47.295512+0000","last_became_active":"2024-09-15T02:49:53.384765+0000","last_became_peered":"2024-09-15T02:49:53.384765+0000","last_unstale":"2024-09-15T02:50:47.295512+0000","last_undegraded":"2024-09-15T02:50:47.295512+0000","last_fullsized":"2024-09-15T02:50:47.295512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:43.793858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:43.793858+0000","last_clean_scrub_stamp":"2024-09-15T02:50:43.793858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:55.648817+0000","scrub_duration":0.0045303080000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T02:50:47.066603+0000","last_change":"2024-09-15T02:50:40.476884+0000","last_active":"2024-09-15T02:50:47.066603+0000","last_peered":"2024-09-15T02:50:47.066603+0000","last_clean":"2024-09-15T02:50:47.066603+0000","last_became_active":"2024-09-15T02:48:27.400548+0000","last_became_peered":"2024-09-15T02:48:27.400548+0000","last_unstale":"2024-09-15T02:50:47.066603+0000","last_undegraded":"2024-09-15T02:50:47.066603+0000","last_fullsized":"2024-09-15T02:50:47.066603+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:40.476570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:40.476570+0000","last_clean_scrub_stamp":"2024-09-15T02:50:40.476570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:04.142168+0000","scrub_duration":0.011617722,"objects_trimmed":0,"snaptrim_duration":0.00060531599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"146'3","reported_seq":83,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T02:50:47.481403+0000","last_change":"2024-09-15T02:50:41.206526+0000","last_active":"2024-09-15T02:50:47.481403+0000","last_peered":"2024-09-15T02:50:47.481403+0000","last_clean":"2024-09-15T02:50:47.481403+0000","last_became_active":"2024-09-15T02:50:04.488695+0000","last_became_peered":"2024-09-15T02:50:04.488695+0000","last_unstale":"2024-09-15T02:50:47.481403+0000","last_undegraded":"2024-09-15T02:50:47.481403+0000","last_fullsized":"2024-09-15T02:50:47.481403+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'3","last_scrub_stamp":"2024-09-15T02:50:41.206367+0000","last_deep_scrub":"146'3","last_deep_scrub_stamp":"2024-09-15T02:50:41.206367+0000","last_clean_scrub_stamp":"2024-09-15T02:50:41.206367+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:03.278836+0000","scrub_duration":0.0060310169999999996,"objects_trimmed":0,"snaptrim_duration":0.00081346599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-15T02:50:59.775623+0000","last_change":"2024-09-15T02:50:45.858578+0000","last_active":"2024-09-15T02:50:59.775623+0000","last_peered":"2024-09-15T02:50:59.775623+0000","last_clean":"2024-09-15T02:50:59.775623+0000","last_became_active":"2024-09-15T02:48:27.403491+0000","last_became_peered":"2024-09-15T02:48:27.403491+0000","last_unstale":"2024-09-15T02:50:59.775623+0000","last_undegraded":"2024-09-15T02:50:59.775623+0000","last_fullsized":"2024-09-15T02:50:59.775623+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:45.858270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:45.858270+0000","last_clean_scrub_stamp":"2024-09-15T02:50:45.858270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:58.551274+0000","scrub_duration":0.0037072379999999999,"objects_trimmed":0,"snaptrim_duration":0.00070136700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":83,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T02:50:47.481401+0000","last_change":"2024-09-15T02:50:39.190558+0000","last_active":"2024-09-15T02:50:47.481401+0000","last_peered":"2024-09-15T02:50:47.481401+0000","last_clean":"2024-09-15T02:50:47.481401+0000","last_became_active":"2024-09-15T02:50:04.612908+0000","last_became_peered":"2024-09-15T02:50:04.612908+0000","last_unstale":"2024-09-15T02:50:47.481401+0000","last_undegraded":"2024-09-15T02:50:47.481401+0000","last_fullsized":"2024-09-15T02:50:47.481401+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T02:50:39.190360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T02:50:39.190360+0000","last_clean_scrub_stamp":"2024-09-15T02:50:39.190360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:41.335481+0000","scrub_duration":0.0036948229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"147'2","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-15T02:50:59.077015+0000","last_change":"2024-09-15T02:50:46.781118+0000","last_active":"2024-09-15T02:50:59.077015+0000","last_peered":"2024-09-15T02:50:59.077015+0000","last_clean":"2024-09-15T02:50:59.077015+0000","last_became_active":"2024-09-15T02:48:28.347123+0000","last_became_peered":"2024-09-15T02:48:28.347123+0000","last_unstale":"2024-09-15T02:50:59.077015+0000","last_undegraded":"2024-09-15T02:50:59.077015+0000","last_fullsized":"2024-09-15T02:50:59.077015+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'2","last_scrub_stamp":"2024-09-15T02:50:46.780771+0000","last_deep_scrub":"147'2","last_deep_scrub_stamp":"2024-09-15T02:50:46.780771+0000","last_clean_scrub_stamp":"2024-09-15T02:50:46.780771+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:59.135528+0000","scrub_duration":0.0072550189999999997,"objects_trimmed":0,"snaptrim_duration":0.00071019300000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"147'114","reported_seq":292,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-15T02:50:59.076926+0000","last_change":"2024-09-15T02:50:45.812805+0000","last_active":"2024-09-15T02:50:59.076926+0000","last_peered":"2024-09-15T02:50:59.076926+0000","last_clean":"2024-09-15T02:50:59.076926+0000","last_became_active":"2024-09-15T02:48:24.407146+0000","last_became_peered":"2024-09-15T02:48:24.407146+0000","last_unstale":"2024-09-15T02:50:59.076926+0000","last_undegraded":"2024-09-15T02:50:59.076926+0000","last_fullsized":"2024-09-15T02:50:59.076926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'114","last_scrub_stamp":"2024-09-15T02:50:45.812597+0000","last_deep_scrub":"147'114","last_deep_scrub_stamp":"2024-09-15T02:50:45.812597+0000","last_clean_scrub_stamp":"2024-09-15T02:50:45.812597+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:45.930680+0000","scrub_duration":0.010723233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15210,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":98,"seq":420906795027,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300236,"kb_used_data":1728,"kb_used_omap":11,"kb_used_meta":298484,"kb_avail":94071604,"statfs":{"total":96636764160,"available":96329322496,"internally_reserved":0,"allocated":1769472,"data_stored":1337929,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":12037,"internal_metadata":305647867},"hb_peers":[1,3,4,5,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.75800000000000001},{"interface":"front","average":{"1min":57.097000000000001,"5min":57.097000000000001,"15min":57.097000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":951.47400000000005,"5min":951.47400000000005,"15min":951.47400000000005},"last":1.032}]},{"osd":3,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.88},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.93500000000000005}]},{"osd":4,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.353},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.91000000000000003}]},{"osd":5,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":940.476}]},{"osd":7,"last update":"Sun Sep 15 02:50:47 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.2909999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":33.119,"5min":33.119,"15min":33.119},"min":{"1min":1.173,"5min":1.173,"15min":1.173},"max":{"1min":571.68600000000004,"5min":571.68600000000004,"15min":571.68600000000004},"last":1.246},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.0589999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":10,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.109},{"interface":"front","average":{"1min":8.718,"5min":8.718,"15min":8.718},"min":{"1min":1.2,"5min":1.2,"15min":1.2},"max":{"1min":130.244,"5min":130.244,"15min":130.244},"last":1.4259999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.3240000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":1.248,"5min":1.248,"15min":1.248},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.3759999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"min":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":1.286,"5min":1.286,"15min":1.286},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.55}]}]},{"osd":7,"up_from":22,"seq":94489280550,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.3049999999999999,"15min":1.3660000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.476,"5min":11.964,"15min":11.964},"last":0.92000000000000004},{"interface":"front","average":{"1min":30.149999999999999,"5min":6.6879999999999997,"15min":2.778},"min":{"1min":0.75700000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":523.76099999999997,"5min":523.76099999999997,"15min":523.76099999999997},"last":1.0820000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":27.459,"5min":6.3040000000000003,"15min":2.778},"min":{"1min":0.872,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":502.36000000000001,"5min":502.36000000000001,"15min":502.36000000000001},"last":1.04},{"interface":"front","average":{"1min":31.748999999999999,"5min":7.1909999999999998,"15min":3.0979999999999999},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":502.52999999999997,"5min":502.52999999999997,"15min":502.52999999999997},"last":1.4159999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.997},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.121}]},{"osd":6,"last update":"Sun Sep 15 02:50:44 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.522}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":39.049999999999997,"5min":8.6910000000000007,"15min":3.6309999999999998},"min":{"1min":0.97399999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":754.66999999999996,"5min":754.66999999999996,"15min":754.66999999999996},"last":1.593},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.2829999999999999,"15min":1.2569999999999999},"min":{"1min":1.0529999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.756,"5min":1.829,"15min":1.829},"last":1.649}]},{"osd":9,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.321,"15min":1.3169999999999999},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7010000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.621},{"interface":"front","average":{"1min":1.141,"5min":62.872999999999998,"15min":73.162000000000006},"min":{"1min":0.75800000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5189999999999999,"5min":934.447,"15min":934.447},"last":1.1859999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":11.114000000000001,"15min":12.766999999999999},"min":{"1min":0.86199999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5369999999999999,"5min":287.02499999999998,"15min":287.02499999999998},"last":1.155},{"interface":"front","average":{"1min":1.393,"5min":1.2250000000000001,"15min":1.1970000000000001},"min":{"1min":0.97899999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.3260000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":13,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.98999999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7,"5min":1.829,"15min":1.829},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.542,"5min":1.4450000000000001,"15min":1.429},"min":{"1min":1.2230000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.869,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.726}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.385,"15min":1.405},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.583,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":1.496},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.419,"15min":1.411},"min":{"1min":1.0009999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.5609999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":15.961,"5min":4.359,"15min":2.4249999999999998},"min":{"1min":1.02,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":290.47899999999998,"5min":290.47899999999998,"15min":290.47899999999998},"last":1.635},{"interface":"front","average":{"1min":1.605,"5min":1.573,"15min":1.5680000000000001},"min":{"1min":1.2290000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.966,"5min":4.2679999999999998,"15min":4.2679999999999998},"last":1.7569999999999999}]}]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303252,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":38.073999999999998,"5min":48.182000000000002,"15min":49.866999999999997},"min":{"1min":0.623,"5min":0.439,"15min":0.439},"max":{"1min":706.45600000000002,"5min":743.34100000000001,"15min":743.34100000000001},"last":1.125},{"interface":"front","average":{"1min":12.116,"5min":3.1520000000000001,"15min":1.6579999999999999},"min":{"1min":0.60699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":213.25700000000001,"5min":213.25700000000001,"15min":213.25700000000001},"last":0.91500000000000004}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.68200000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4970000000000001,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.052,"15min":1.024},"min":{"1min":0.85499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.99399999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":1.071},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.76800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3999999999999999,"5min":4.141,"15min":4.141},"last":0.83799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":45.929000000000002,"5min":10.048,"15min":4.0679999999999996},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":849.30799999999999,"5min":849.30799999999999,"15min":849.30799999999999},"last":1.248},{"interface":"front","average":{"1min":1.083,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.69099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6020000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.72999999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":65.325000000000003,"15min":75.989000000000004},"min":{"1min":0.84799999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8089999999999999,"5min":936.86800000000005,"15min":936.86800000000005},"last":1.3140000000000001},{"interface":"front","average":{"1min":66.593000000000004,"5min":38.953000000000003,"15min":34.347000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":729.71000000000004,"5min":729.71000000000004,"15min":729.71000000000004},"last":0.80400000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.3260000000000001},"min":{"1min":0.92300000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.867,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.349},{"interface":"front","average":{"1min":1.25,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.629,"5min":2.262,"15min":2.262},"last":0.95699999999999996}]},{"osd":12,"last update":"Sun Sep 15 02:50:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":31.972000000000001,"15min":37.112000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5960000000000001,"5min":807.56899999999996,"15min":807.56899999999996},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.399,"15min":1.393},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.448}]},{"osd":13,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.21}]},{"osd":14,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":7.1879999999999997,"5min":7.1879999999999997,"15min":7.1879999999999997},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":107.42400000000001,"5min":107.42400000000001,"15min":107.42400000000001},"last":1.159},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.0369999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.361,"15min":1.357},"min":{"1min":0.90200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.776,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.452,"15min":1.4259999999999999},"min":{"1min":1.0669999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.2849999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313252,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":1769472,"data_stored":1337929,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.413,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.66700000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3089999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.97899999999999998}]},{"osd":3,"last update":"Sun Sep 15 02:50:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.155},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":1.117}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.78300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1819999999999999,"15min":1.1699999999999999},"min":{"1min":0.80900000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7589999999999999,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.1779999999999999}]},{"osd":6,"last update":"Sun Sep 15 02:50:45 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":8,"last update":"Sun Sep 15 02:49:27 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.069},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.099}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":6.899,"5min":2.3279999999999998,"15min":1.5660000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":106.413,"5min":106.413,"15min":106.413},"last":1.139},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4950000000000001,"15min":1.5369999999999999},"min":{"1min":0.82399999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.7869999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.421}]},{"osd":10,"last update":"Sun Sep 15 02:50:49 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.224,"15min":1.202},"min":{"1min":1.034,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.4330000000000001,"15min":1.4350000000000001},"min":{"1min":1.2370000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.6759999999999999,"5min":2.46,"15min":2.46},"last":1.4910000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":42.451000000000001,"5min":9.5399999999999991,"15min":4.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":740.58100000000002,"5min":740.58100000000002,"15min":740.58100000000002},"last":1.262},{"interface":"front","average":{"1min":1.389,"5min":1.357,"15min":1.351},"min":{"1min":1.0029999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.579,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.351}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.585,"15min":1.617},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.74,"5min":3.363,"15min":3.363},"last":1.468},{"interface":"front","average":{"1min":1.536,"5min":1.4550000000000001,"15min":1.4410000000000001},"min":{"1min":1.073,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.536}]},{"osd":13,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.5149999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:45 2024","interfaces":[{"interface":"back","average":{"1min":70.560000000000002,"5min":53.128999999999998,"15min":50.223999999999997},"min":{"1min":1.0880000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":986.84799999999996,"5min":986.84799999999996,"15min":986.84799999999996},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.544,"15min":1.522},"min":{"1min":1.3580000000000001,"5min":1.016,"15min":1.016},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.657}]}]},{"osd":0,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1769472,"data_stored":1337929,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0820000000000001,"15min":1.107},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.1910000000000001,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":1.006},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0109999999999999,"15min":1.0049999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.91500000000000004}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.016,"15min":0.998},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4630000000000001,"5min":2.851,"15min":2.851},"last":1.226},{"interface":"front","average":{"1min":1.1890000000000001,"5min":34.731000000000002,"15min":40.320999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.8360000000000001,"5min":503.59300000000002,"15min":503.59300000000002},"last":1.3069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.214,"15min":1.21},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.0840000000000001,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.127,"5min":8.8989999999999991,"15min":10.195},"min":{"1min":0.78500000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.5580000000000001,"5min":223.92500000000001,"15min":223.92500000000001},"last":0.96099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.46,"15min":1.444},"min":{"1min":0.95699999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.403,"5min":3.431,"15min":3.431},"last":1.405},{"interface":"front","average":{"1min":1.653,"5min":34.695999999999998,"15min":40.203000000000003},"min":{"1min":1.008,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4660000000000002,"5min":954.81700000000001,"15min":954.81700000000001},"last":1.5660000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":2.1219999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":2.714}]},{"osd":10,"last update":"Sun Sep 15 02:50:00 2024","interfaces":[{"interface":"back","average":{"1min":44.524000000000001,"5min":44.524000000000001,"15min":44.524000000000001},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":860.30600000000004,"5min":860.30600000000004,"15min":860.30600000000004},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.3600000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.397},"min":{"1min":0.91000000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.651,"5min":23.751000000000001,"15min":27.434000000000001},"min":{"1min":1.133,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.75,"5min":549.39099999999996,"15min":549.39099999999996},"last":1.702}]},{"osd":12,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":31.309000000000001,"15min":36.253999999999998},"min":{"1min":0.90200000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.508,"5min":893.91999999999996,"15min":893.91999999999996},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.762,"5min":1.694,"15min":1.6819999999999999},"min":{"1min":1.1679999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.986}]},{"osd":13,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":5.6760000000000002,"15min":6.3259999999999996},"min":{"1min":1.1100000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8079999999999998,"5min":122.18000000000001,"15min":122.18000000000001},"last":2.012},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.3320000000000001,"15min":1.3180000000000001},"min":{"1min":0.83899999999999997,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.9179999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":6.657,"5min":11.952999999999999,"15min":12.835000000000001},"min":{"1min":0.94299999999999995,"5min":0.747,"15min":0.747},"max":{"1min":85.116,"5min":286.791,"15min":286.791},"last":1.669},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.478,"15min":1.4810000000000001},"min":{"1min":1.0049999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.2000000000000002,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.7390000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":18.294,"15min":21.045000000000002},"min":{"1min":1.046,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.367,"5min":504.36000000000001,"15min":504.36000000000001},"last":1.603},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.607,"15min":1.593},"min":{"1min":0.92100000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4300000000000002,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":1.7969999999999999}]}]},{"osd":13,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.444},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.369},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.546}]},{"osd":3,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.4650000000000001,"15min":1.4710000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.899,"5min":1.982,"15min":1.982},"last":0.96599999999999997},{"interface":"front","average":{"1min":49.173999999999999,"5min":11.083,"15min":4.7350000000000003},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":764.72699999999998,"5min":764.72699999999998,"15min":764.72699999999998},"last":0.92300000000000004}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4450000000000001,"15min":1.458},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.353},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.6399999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0169999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.657},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.78}]},{"osd":6,"last update":"Sun Sep 15 02:50:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.286},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3140000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"min":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":2.294},{"interface":"front","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.597}]},{"osd":8,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.643,"15min":1.6990000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.994,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.7589999999999999,"15min":1.7969999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.5699999999999998,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":1.476}]},{"osd":9,"last update":"Sun Sep 15 02:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.6950000000000001,"15min":1.7450000000000001},"min":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"max":{"1min":1.9510000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.9730000000000001,"15min":2.0670000000000002},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.9650000000000001,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.4470000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:51:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":21.891999999999999,"15min":25.334},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":1.4610000000000001,"5min":501.60599999999999,"15min":501.60599999999999},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.988,"15min":2.1000000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":1.5089999999999999,"5min":2.702,"15min":2.702},"last":1.341}]},{"osd":11,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.8420000000000001,"15min":1.8640000000000001},"min":{"1min":1.111,"5min":1.111,"15min":1.111},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":1.411},{"interface":"front","average":{"1min":1.746,"5min":2.0179999999999998,"15min":2.0640000000000001},"min":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"max":{"1min":2.4180000000000001,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5840000000000001}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.833,"15min":1.8280000000000001},"min":{"1min":1.228,"5min":1.228,"15min":1.228},"max":{"1min":2.4020000000000001,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.524},{"interface":"front","average":{"1min":54.465000000000003,"5min":12.493,"15min":5.4969999999999999},"min":{"1min":1.333,"5min":1.333,"15min":1.333},"max":{"1min":841.86500000000001,"5min":841.86500000000001,"15min":841.86500000000001},"last":1.7170000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.883,"15min":1.893},"min":{"1min":1.284,"5min":1.284,"15min":1.284},"max":{"1min":2.4350000000000001,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.9370000000000001,"15min":1.9379999999999999},"min":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"max":{"1min":2.6339999999999999,"5min":3.069,"15min":3.069},"last":1.641}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":31.495999999999999,"15min":36.398000000000003},"min":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"max":{"1min":2.8380000000000001,"5min":663.61500000000001,"15min":663.61500000000001},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.879,"5min":1.8640000000000001,"15min":1.8620000000000001},"min":{"1min":1.369,"5min":1.369,"15min":1.369},"max":{"1min":2.5,"5min":2.5,"15min":2.5},"last":1.6899999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":41.189,"5min":9.5060000000000002,"15min":4.226},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":766.90200000000004,"5min":766.90200000000004,"15min":766.90200000000004},"last":0.746},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.96299999999999997},"min":{"1min":0.64800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.204,"5min":2.081,"15min":2.081},"last":1.1319999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1419999999999999,"15min":1.147},"min":{"1min":0.876,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.599,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.248,"15min":1.27},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.681,"5min":2.879,"15min":2.879},"last":1.0469999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.276,"15min":1.286},"min":{"1min":0.76900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.841,"5min":2.202,"15min":2.202},"last":1.25},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.288,"15min":1.3100000000000001},"min":{"1min":0.85599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6479999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.93200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":483.48599999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.4199999999999999,"15min":1.4359999999999999},"min":{"1min":0.872,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7090000000000001,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.427,"15min":1.456},"min":{"1min":0.94599999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.673,"5min":4.0119999999999996,"15min":4.0119999999999996},"last":1.23}]},{"osd":9,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.6699999999999999,"15min":1.7050000000000001},"min":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":1.8819999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.407,"5min":1.6970000000000001,"15min":1.7450000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.7310000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.1899999999999999}]},{"osd":10,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":50.637999999999998,"5min":50.637999999999998,"15min":50.637999999999998},"min":{"1min":1.125,"5min":1.125,"15min":1.125},"max":{"1min":985.39800000000002,"5min":985.39800000000002,"15min":985.39800000000002},"last":1.21},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.3839999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:40 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":10.401,"15min":11.865},"min":{"1min":1.1930000000000001,"5min":1.038,"15min":1.038},"max":{"1min":2.0259999999999998,"5min":189.79599999999999,"15min":189.79599999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.284,"5min":9.0129999999999999,"15min":10.301},"min":{"1min":0.878,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.8,"5min":163.952,"15min":163.952},"last":1.0880000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.538,"15min":1.548},"min":{"1min":1.137,"5min":1.052,"15min":1.052},"max":{"1min":1.95,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.268},{"interface":"front","average":{"1min":1.335,"5min":1.4550000000000001,"15min":1.4750000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6859999999999999,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.157}]},{"osd":14,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.677,"15min":1.6919999999999999},"min":{"1min":1.3120000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.101,"5min":4.4119999999999999,"15min":4.4119999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.726,"15min":1.752},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.1360000000000001,"5min":3.839,"15min":3.839},"last":1.486}]},{"osd":15,"last update":"Sun Sep 15 02:50:44 2024","interfaces":[{"interface":"back","average":{"1min":16.702000000000002,"5min":27.265999999999998,"15min":29.026},"min":{"1min":1.52,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":331.16500000000002,"5min":510.72500000000002,"15min":510.72500000000002},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.7669999999999999,"15min":1.7889999999999999},"min":{"1min":1.3,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.0390000000000001,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.353}]}]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303252,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":27.241,"5min":6.2729999999999997,"15min":2.7789999999999999},"min":{"1min":0.80700000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":472.28100000000001,"5min":472.28100000000001,"15min":472.28100000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.3500000000000001,"5min":2.113,"15min":2.113},"last":0.86199999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":39.546999999999997,"5min":8.6379999999999999,"15min":3.4860000000000002},"min":{"1min":0.62,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":457.49000000000001,"5min":457.49000000000001,"15min":457.49000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.046,"15min":1.044},"min":{"1min":0.80000000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.2}]},{"osd":3,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1759999999999999,"15min":1.1970000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.573,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.1359999999999999,"5min":9.2140000000000004,"15min":10.56},"min":{"1min":0.78200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5129999999999999,"5min":183.30199999999999,"15min":183.30199999999999},"last":1.101}]},{"osd":4,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":31.363,"15min":36.399999999999999},"min":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5249999999999999,"5min":681.39599999999996,"15min":681.39599999999996},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":42.091999999999999,"15min":48.93},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.554,"5min":921.02099999999996,"15min":921.02099999999996},"last":1.464}]},{"osd":5,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.325,"15min":1.337},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":2.504,"15min":2.504},"last":1.0660000000000001},{"interface":"front","average":{"1min":80.025999999999996,"5min":17.149999999999999,"15min":6.6710000000000003},"min":{"1min":0.86799999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":809.87800000000004,"5min":809.87800000000004,"15min":809.87800000000004},"last":1.2829999999999999}]},{"osd":9,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.337,"15min":1.323},"min":{"1min":1.0589999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.163},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4159999999999999,"15min":1.4179999999999999},"min":{"1min":1.0089999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.929,"5min":2.089,"15min":2.089},"last":1.3480000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2649999999999999,"15min":1.266},"min":{"1min":0.93500000000000005,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.518},{"interface":"front","average":{"1min":82.028999999999996,"5min":17.399999999999999,"15min":6.6280000000000001},"min":{"1min":1.1879999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1464.03,"5min":1464.03,"15min":1464.03},"last":1.391}]},{"osd":11,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.5189999999999999,"15min":1.5269999999999999},"min":{"1min":1.0349999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":1.925,"5min":3.133,"15min":3.133},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.478,"15min":1.46},"min":{"1min":1.288,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.0590000000000002,"5min":2.077,"15min":2.077},"last":1.4259999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.524,"15min":1.5249999999999999},"min":{"1min":1.1220000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.9710000000000001,"5min":2.468,"15min":2.468},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.587,"15min":1.5960000000000001},"min":{"1min":1.0840000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.9379999999999999,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.6459999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:25 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.524,"15min":1.5049999999999999},"min":{"1min":1.349,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.55},{"interface":"front","average":{"1min":1.5940000000000001,"5min":3.012,"15min":3.2480000000000002},"min":{"1min":1.262,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.1419999999999999,"5min":34.311999999999998,"15min":34.311999999999998},"last":1.5780000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303300,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068540,"statfs":{"total":96636764160,"available":96326184960,"internally_reserved":0,"allocated":1826816,"data_stored":1867435,"data_compressed":261683,"data_compressed_allocated":557056,"data_compressed_original":1560576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.93799999999999994,"15min":0.92600000000000005},"min":{"1min":0.623,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.753},{"interface":"front","average":{"1min":1.091,"5min":24.271999999999998,"15min":28.135999999999999},"min":{"1min":0.76400000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.762,"5min":611.07899999999995,"15min":611.07899999999995},"last":0.83299999999999996}]},{"osd":1,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":37.683,"15min":43.805},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5069999999999999,"5min":917.20299999999997,"15min":917.20299999999997},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93400000000000005,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.008}]},{"osd":3,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.109,"15min":1.1040000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.121,"15min":1.121},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.452,"5min":1.649,"15min":1.649},"last":1.123}]},{"osd":4,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4119999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.089,"15min":1.093},"min":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.5009999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.05}]},{"osd":5,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.274,"15min":1.278},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6830000000000001,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.149},{"interface":"front","average":{"1min":19.334,"5min":4.782,"15min":2.3570000000000002},"min":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":381.56099999999998,"5min":381.56099999999998,"15min":381.56099999999998},"last":1.091}]},{"osd":6,"last update":"Sun Sep 15 02:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.171},{"interface":"front","average":{"1min":1.403,"5min":1.403,"15min":1.403},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.98799999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:23 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.2050000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.492},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":30.053000000000001,"5min":30.053000000000001,"15min":30.053000000000001},"last":0.94899999999999995}]},{"osd":9,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":31.562000000000001,"5min":7.5309999999999997,"15min":3.5259999999999998},"min":{"1min":1.194,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":631.01599999999996,"5min":631.01599999999996,"15min":631.01599999999996},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5629999999999999,"15min":1.571},"min":{"1min":0.96199999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.905,"5min":2.173,"15min":2.173},"last":64.254999999999995}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":22.736999999999998,"15min":26.260000000000002},"min":{"1min":1.091,"5min":0.745,"15min":0.745},"max":{"1min":2.5259999999999998,"5min":531.54300000000001,"15min":531.54300000000001},"last":1.369},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.603,"15min":1.615},"min":{"1min":0.93899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.8260000000000001,"5min":2.802,"15min":2.802},"last":1.2869999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.425,"15min":1.4299999999999999},"min":{"1min":0.83899999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.677,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.262},{"interface":"front","average":{"1min":38.649999999999999,"5min":9.0510000000000002,"15min":4.1180000000000003},"min":{"1min":1.2390000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":741.423,"5min":741.423,"15min":741.423},"last":1.431}]}]},{"osd":15,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,5,6,7,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.998,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.5720000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.042,"5min":0.95099999999999996,"15min":0.93600000000000005},"min":{"1min":0.86899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2470000000000001,"5min":1.331,"15min":1.331},"last":0.86299999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.296},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.397}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":40.598999999999997,"5min":8.9719999999999995,"15min":3.7010000000000001},"min":{"1min":0.85699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":867.53899999999999,"5min":867.53899999999999,"15min":867.53899999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.544},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.946}]},{"osd":5,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":0.89600000000000002}]},{"osd":6,"last update":"Sun Sep 15 02:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":1.002,"5min":1.002,"15min":1.002},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.232},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.141}]},{"osd":7,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.206},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.1619999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:47 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":18.454999999999998,"15min":21.263000000000002},"min":{"1min":1.161,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.5049999999999999,"5min":488.12900000000002,"15min":488.12900000000002},"last":1.464},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.425,"15min":1.383},"min":{"1min":1.2310000000000001,"5min":0.876,"15min":0.876},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.4890000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":11,"last update":"Sun Sep 15 02:50:31 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.6279999999999999,"15min":1.597},"min":{"1min":1.363,"5min":1.085,"15min":1.085},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.49,"15min":1.462},"min":{"1min":1.1970000000000001,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.2569999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":5.4729999999999999,"15min":6.101},"min":{"1min":1.169,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2850000000000001,"5min":121.304,"15min":121.304},"last":1.357},{"interface":"front","average":{"1min":1.78,"5min":1.516,"15min":1.472},"min":{"1min":1.3009999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.294}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.73,"15min":1.694},"min":{"1min":1.4970000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"max":{"1min":2.5939999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.609},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.605,"15min":1.577},"min":{"1min":1.256,"5min":1.093,"15min":1.093},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.3899999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.6120000000000001,"15min":1.5760000000000001},"min":{"1min":1.3440000000000001,"5min":1.26,"15min":1.26},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.534},{"interface":"front","average":{"1min":1.8400000000000001,"5min":14.928000000000001,"15min":17.109999999999999},"min":{"1min":1.361,"5min":1.141,"15min":1.141},"max":{"1min":2.4620000000000002,"5min":383.084,"15min":383.084},"last":1.4450000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":38.845999999999997,"5min":38.845999999999997,"15min":38.845999999999997},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":684.24900000000002,"5min":684.24900000000002,"15min":684.24900000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":28.260000000000002,"5min":6.4320000000000004,"15min":2.794},"min":{"1min":0.752,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":463.55599999999998,"5min":463.55599999999998,"15min":463.55599999999998},"last":0.99399999999999999},{"interface":"front","average":{"1min":36.347999999999999,"5min":41.273000000000003,"15min":42.094000000000001},"min":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":600.65999999999997,"5min":913.93899999999996,"15min":913.93899999999996},"last":1.165}]},{"osd":4,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.244,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.93899999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2130000000000001,"15min":1.2010000000000001},"min":{"1min":0.94999999999999996,"5min":0.747,"15min":0.747},"max":{"1min":1.681,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.103},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.214,"15min":1.2130000000000001},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.3400000000000001}]},{"osd":6,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":62.308,"5min":62.308,"15min":62.308},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":917.43399999999997,"5min":917.43399999999997,"15min":917.43399999999997},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.2410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.131}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":11,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.468}]},{"osd":12,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":0.96599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":1.752,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":1.3640000000000001},{"interface":"front","average":{"1min":10.009,"5min":3.2250000000000001,"15min":2.0950000000000002},"min":{"1min":1.1870000000000001,"5min":1.024,"15min":1.024},"max":{"1min":145.501,"5min":145.501,"15min":145.501},"last":1.526}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.575,"15min":1.577},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.96,"5min":1.966,"15min":1.966},"last":1.554},{"interface":"front","average":{"1min":1.389,"5min":4.6719999999999997,"15min":5.2190000000000003},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.641,"5min":92.698999999999998,"15min":92.698999999999998},"last":1.2769999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":1.367,"5min":1.367,"15min":1.367},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.597}]}]},{"osd":8,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1338526,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":1,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.6950000000000001,"15min":1.823},"min":{"1min":0.64100000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.534,"5min":27.010999999999999,"15min":27.010999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":63.902000000000001,"5min":31.067,"15min":25.594999999999999},"min":{"1min":0.61499999999999999,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":979.23099999999999,"5min":979.23099999999999,"15min":979.23099999999999},"last":0.66300000000000003}]},{"osd":2,"last update":"Sun Sep 15 02:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1339999999999999,"15min":1.119},"min":{"1min":0.95799999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":828.08299999999997},{"interface":"front","average":{"1min":1.278,"5min":11.226000000000001,"15min":12.884},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6279999999999999,"5min":276.096,"15min":276.096},"last":1.1359999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.014,"15min":1.018},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.464},{"interface":"front","average":{"1min":1.226,"5min":1.1200000000000001,"15min":1.1020000000000001},"min":{"1min":0.77200000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":1.081}]},{"osd":6,"last update":"Sun Sep 15 02:50:45 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.218},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.96699999999999997}]},{"osd":7,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":40.927999999999997,"5min":16.664999999999999,"15min":12.621},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":918.68899999999996,"5min":918.68899999999996,"15min":918.68899999999996},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.68200000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.4610000000000001,"5min":1.675,"15min":1.675},"last":1.337}]},{"osd":9,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.3120000000000001},"min":{"1min":0.85699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.5509999999999999,"5min":2.331,"15min":2.331},"last":1.577},{"interface":"front","average":{"1min":33.042000000000002,"5min":41.652000000000001,"15min":43.087000000000003},"min":{"1min":1.133,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":487.935,"5min":849.11599999999999,"15min":849.11599999999999},"last":1.802}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.24,"15min":1.2649999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.5589999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":1.248},{"interface":"front","average":{"1min":19.361999999999998,"5min":5.0179999999999998,"15min":2.6280000000000001},"min":{"1min":0.88,"5min":0.751,"15min":0.751},"max":{"1min":414.16399999999999,"5min":414.16399999999999,"15min":414.16399999999999},"last":1.7430000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":34.908000000000001,"15min":37.692999999999998},"min":{"1min":1.036,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":387.28899999999999,"5min":423.70800000000003,"15min":423.70800000000003},"last":1.29},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3120000000000001,"15min":1.3380000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5049999999999999,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.3839999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.5189999999999999},"min":{"1min":1.2509999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.7869999999999999,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4390000000000001,"15min":1.454},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9770000000000001,"5min":3.012,"15min":3.012},"last":1.4139999999999999}]},{"osd":15,"last update":"Sun Sep 15 02:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.496,"15min":1.504},"min":{"1min":0.97499999999999998,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8560000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.893},{"interface":"front","average":{"1min":1.5740000000000001,"5min":26.928000000000001,"15min":31.154},"min":{"1min":1.196,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":1.8979999999999999,"5min":603.77599999999995,"15min":603.77599999999995},"last":1.5189999999999999}]}]},{"osd":9,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1337948,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,6,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.878}]},{"osd":1,"last update":"Sun Sep 15 02:50:19 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.84799999999999998},{"interface":"front","average":{"1min":27.036000000000001,"5min":27.036000000000001,"15min":27.036000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":440.40699999999998,"5min":440.40699999999998,"15min":440.40699999999998},"last":1.28}]},{"osd":2,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.1459999999999999,"15min":1.034},"min":{"1min":0.76400000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":13.266,"5min":13.266,"15min":13.266},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.044,"15min":1.034},"min":{"1min":0.85999999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4419999999999999,"5min":1.74,"15min":1.74},"last":1.018}]},{"osd":3,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":14.573,"5min":3.7749999999999999,"15min":1.9750000000000001},"min":{"1min":0.93899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":215.09100000000001,"5min":215.09100000000001,"15min":215.09100000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.0840000000000001,"15min":1.05},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Sun Sep 15 02:50:52 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.1659999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":8,"last update":"Sun Sep 15 02:50:35 2024","interfaces":[{"interface":"back","average":{"1min":24.364999999999998,"5min":5.8689999999999998,"15min":2.786},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":373.05500000000001,"5min":373.05500000000001,"15min":373.05500000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1819999999999999,"15min":1.196},"min":{"1min":0.745,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8240000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.45}]},{"osd":10,"last update":"Sun Sep 15 02:50:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.24,"15min":1.2170000000000001},"min":{"1min":1.0800000000000001,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.2989999999999999,"15min":1.278},"min":{"1min":1.1619999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7769999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.1970000000000001}]},{"osd":11,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.349,"15min":1.335},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3260000000000001,"15min":1.3120000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0219999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.252}]},{"osd":12,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":49.465000000000003,"5min":10.974,"15min":4.5590000000000002},"min":{"1min":1.111,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":769.28800000000001,"5min":769.28800000000001,"15min":769.28800000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.296,"15min":1.276},"min":{"1min":0.93500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8620000000000001,"5min":2.226,"15min":2.226},"last":1.5429999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":14,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.6120000000000001},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.0510000000000002,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.425,"15min":1.4079999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.0920000000000001,"5min":2.347,"15min":2.347},"last":1.589}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":10,"up_from":42,"seq":180388626462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299604,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":297854,"kb_avail":94072236,"statfs":{"total":96636764160,"available":96329969664,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1784,"internal_metadata":305002760},"hb_peers":[0,1,3,4,5,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.177,"15min":1.2090000000000001},"min":{"1min":0.76500000000000001,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.2529999999999999,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":56.142000000000003,"5min":12.061999999999999,"15min":4.7160000000000002},"min":{"1min":0.82699999999999996,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":688.75300000000004,"5min":688.75300000000004,"15min":688.75300000000004},"last":1.1040000000000001}]},{"osd":1,"last update":"Sun Sep 15 02:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.70599999999999996,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.323,"5min":1.954,"15min":1.954},"last":1.145},{"interface":"front","average":{"1min":3.2400000000000002,"5min":1.4490000000000001,"15min":1.151},"min":{"1min":0.93300000000000005,"5min":0.626,"15min":0.626},"max":{"1min":39.012999999999998,"5min":39.012999999999998,"15min":39.012999999999998},"last":1.294}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Sep 15 02:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":9.8949999999999996,"15min":11.361000000000001},"min":{"1min":0.747,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5529999999999999,"5min":189.78700000000001,"15min":189.78700000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":32.832999999999998,"5min":7.4459999999999997,"15min":3.2149999999999999},"min":{"1min":0.96499999999999997,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":571.23099999999999,"5min":571.23099999999999,"15min":571.23099999999999},"last":1.3899999999999999}]},{"osd":4,"last update":"Sun Sep 15 02:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3260000000000001,"15min":1.321},"min":{"1min":0.86799999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.708,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.123,"15min":1.111},"min":{"1min":0.85699999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.4319999999999999}]},{"osd":5,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.5209999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:20 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.552},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.7709999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.472},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.885}]},{"osd":9,"last update":"Sun Sep 15 02:50:56 2024","interfaces":[{"interface":"back","average":{"1min":5.282,"5min":2.3929999999999998,"15min":1.911},"min":{"1min":1.173,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":72.106999999999999,"5min":72.106999999999999,"15min":72.106999999999999},"last":1.796},{"interface":"front","average":{"1min":1.571,"5min":1.468,"15min":1.4510000000000001},"min":{"1min":1.1259999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.831,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.9379999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":9.1959999999999997,"15min":10.494999999999999},"min":{"1min":1.097,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.853,"5min":170.46600000000001,"15min":170.46600000000001},"last":1.5800000000000001},{"interface":"front","average":{"1min":88.305000000000007,"5min":31.451000000000001,"15min":21.975000000000001},"min":{"1min":1.2470000000000001,"5min":0.87,"15min":0.87},"max":{"1min":842.31899999999996,"5min":842.31899999999996,"15min":842.31899999999996},"last":1.7130000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":13,"last update":"Sun Sep 15 02:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.587,"15min":1.583},"min":{"1min":1.1499999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.9099999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"last":1.913},{"interface":"front","average":{"1min":61.734000000000002,"5min":40.506999999999998,"15min":36.969000000000001},"min":{"1min":1.1819999999999999,"5min":1.042,"15min":1.042},"max":{"1min":666.36099999999999,"5min":666.36099999999999,"15min":666.36099999999999},"last":1.6599999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280548,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303304,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068536,"statfs":{"total":96636764160,"available":96326180864,"internally_reserved":0,"allocated":1826816,"data_stored":1867435,"data_compressed":261683,"data_compressed_allocated":557056,"data_compressed_original":1560576,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.071,"15min":1.0840000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.456,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.82199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5900000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.97999999999999998}]},{"osd":1,"last update":"Sun Sep 15 02:49:26 2024","interfaces":[{"interface":"back","average":{"1min":40.106999999999999,"5min":40.106999999999999,"15min":40.106999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":934.24900000000002,"5min":934.24900000000002,"15min":934.24900000000002},"last":1.244},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.7949999999999999}]},{"osd":2,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":55.905999999999999,"5min":12.077999999999999,"15min":4.7729999999999997},"min":{"1min":0.90500000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":986.80999999999995,"5min":986.80999999999995,"15min":986.80999999999995},"last":1.056},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.292,"15min":1.306},"min":{"1min":0.88200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.5600000000000001,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":1.202}]},{"osd":3,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":5.9850000000000003,"5min":5.9850000000000003,"15min":5.9850000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":116.991,"5min":116.991,"15min":116.991},"last":1.417},{"interface":"front","average":{"1min":24.859000000000002,"5min":24.859000000000002,"15min":24.859000000000002},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":571.38099999999997,"5min":571.38099999999997,"15min":571.38099999999997},"last":1.3580000000000001}]},{"osd":4,"last update":"Sun Sep 15 02:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":3.9180000000000001,"5min":3.9180000000000001,"15min":3.9180000000000001},"last":2.117},{"interface":"front","average":{"1min":1.425,"5min":1.425,"15min":1.425},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":1.978}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Sun Sep 15 02:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.157,"15min":1.157},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.1419999999999999}]},{"osd":7,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.3089999999999999,"15min":1.329},"min":{"1min":0.68500000000000005,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.3919999999999999,"15min":1.3839999999999999},"min":{"1min":0.93500000000000005,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.411}]},{"osd":8,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.47,"15min":1.484},"min":{"1min":0.91400000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.079},{"interface":"front","average":{"1min":1.526,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.88300000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.2290000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:49 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.579,"15min":1.575},"min":{"1min":1.1040000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.236},{"interface":"front","average":{"1min":1.405,"5min":1.4510000000000001,"15min":1.4590000000000001},"min":{"1min":0.96099999999999997,"5min":0.879,"15min":0.879},"max":{"1min":2.2949999999999999,"5min":2.774,"15min":2.774},"last":1.0169999999999999}]},{"osd":12,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.659,"15min":1.649},"min":{"1min":1.2090000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.49},{"interface":"front","average":{"1min":1.5509999999999999,"5min":25.244,"15min":29.193000000000001},"min":{"1min":1.2270000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.198,"5min":713.26999999999998,"15min":713.26999999999998},"last":1.4410000000000001}]},{"osd":13,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":22.937000000000001,"15min":26.486999999999998},"min":{"1min":1.0429999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4470000000000001,"5min":637.95699999999999,"15min":637.95699999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.823,"5min":1.752,"15min":1.74},"min":{"1min":1.409,"5min":0.872,"15min":0.872},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.5669999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.694,"15min":1.698},"min":{"1min":1.1519999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.637},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.913,"15min":1.9550000000000001},"min":{"1min":1.331,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.347,"5min":5.0839999999999996,"15min":5.0839999999999996},"last":1.5900000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":42.085999999999999,"15min":48.789999999999999},"min":{"1min":1.4370000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"max":{"1min":2.524,"5min":586.40499999999997,"15min":586.40499999999997},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.377,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2490000000000001,"5min":2.8559999999999999,"15min":2.8559999999999999},"last":1.55}]}]},{"osd":12,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299088,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072752,"statfs":{"total":96636764160,"available":96330498048,"internally_reserved":0,"allocated":1773568,"data_stored":1338507,"data_compressed":254078,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.27,"15min":1.272},"min":{"1min":0.84599999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.9139999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.2509999999999999,"15min":1.2629999999999999},"min":{"1min":0.94899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5669999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.2909999999999999}]},{"osd":3,"last update":"Sun Sep 15 02:49:22 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.7090000000000001,"5min":3.7090000000000001,"15min":3.7090000000000001},"last":1.266}]},{"osd":4,"last update":"Sun Sep 15 02:49:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2210000000000001},{"interface":"front","average":{"1min":29.57,"5min":29.57,"15min":29.57},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":509.505,"5min":509.505,"15min":509.505},"last":1.0700000000000001}]},{"osd":5,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.405},"min":{"1min":1.0029999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0459999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.202}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4830000000000001,"15min":1.4930000000000001},"min":{"1min":1.0169999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.028,"5min":3.7440000000000002,"15min":3.7440000000000002},"last":1.242},{"interface":"front","average":{"1min":1.419,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":1.075,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.0590000000000002,"5min":2.407,"15min":2.407},"last":1.4079999999999999}]},{"osd":8,"last update":"Sun Sep 15 02:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.512,"15min":1.5589999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.121,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":25.489000000000001,"5min":6.2640000000000002,"15min":3.0600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":460.21899999999999,"5min":460.21899999999999,"15min":460.21899999999999},"last":0.996}]},{"osd":9,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.4430000000000001}]},{"osd":10,"last update":"Sun Sep 15 02:50:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.821},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"min":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.3799999999999999}]},{"osd":11,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":14.083,"5min":4.1310000000000002,"15min":2.472},"min":{"1min":1.2889999999999999,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":173.87799999999999,"5min":173.87799999999999,"15min":173.87799999999999},"last":1.9710000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.6830000000000001,"15min":1.6519999999999999},"min":{"1min":1.359,"5min":1.056,"15min":1.056},"max":{"1min":2.359,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.5109999999999999}]},{"osd":13,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.554,"15min":1.508},"min":{"1min":1.4830000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.331},{"interface":"front","average":{"1min":1.917,"5min":1.7330000000000001,"15min":1.702},"min":{"1min":1.5229999999999999,"5min":0.874,"15min":0.874},"max":{"1min":2.3620000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":2.0009999999999999}]},{"osd":14,"last update":"Sun Sep 15 02:50:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.736,"15min":1.702},"min":{"1min":1.4590000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":2.6120000000000001,"5min":3.819,"15min":3.819},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.794,"5min":12.997999999999999,"15min":14.865},"min":{"1min":1.4179999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.1749999999999998,"5min":285.95499999999998,"15min":285.95499999999998},"last":1.7230000000000001}]},{"osd":15,"last update":"Sun Sep 15 02:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":1.226,"5min":1.226,"15min":1.226},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.8660000000000001},{"interface":"front","average":{"1min":85.462000000000003,"5min":85.462000000000003,"15min":85.462000000000003},"min":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"max":{"1min":815.25999999999999,"5min":815.25999999999999,"15min":815.25999999999999},"last":2.0259999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7605,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T02:51:05.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T02:51:06.279 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T02:51:06.279 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T02:51:06.280 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T02:51:06.280 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.346 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T02:51:06.346 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T02:51:06.346 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.424 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T02:51:06.424 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-15T02:51:06.424 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.512 INFO:tasks.ceph.osd.8:Stopped 2024-09-15T02:51:06.512 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-15T02:51:06.512 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.597 INFO:tasks.ceph.osd.12:Stopped 2024-09-15T02:51:06.597 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T02:51:06.598 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.671 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T02:51:06.671 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T02:51:06.671 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.752 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T02:51:06.752 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-15T02:51:06.753 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.842 INFO:tasks.ceph.osd.9:Stopped 2024-09-15T02:51:06.843 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-15T02:51:06.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.916 INFO:tasks.ceph.osd.13:Stopped 2024-09-15T02:51:06.916 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T02:51:06.916 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:06.985 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T02:51:06.986 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-15T02:51:06.986 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.047 INFO:tasks.ceph.osd.6:Stopped 2024-09-15T02:51:07.048 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-15T02:51:07.048 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.110 INFO:tasks.ceph.osd.10:Stopped 2024-09-15T02:51:07.110 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-15T02:51:07.110 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.189 INFO:tasks.ceph.osd.14:Stopped 2024-09-15T02:51:07.189 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T02:51:07.189 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.276 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T02:51:07.277 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-15T02:51:07.277 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.350 INFO:tasks.ceph.osd.7:Stopped 2024-09-15T02:51:07.350 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-15T02:51:07.350 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.420 INFO:tasks.ceph.osd.11:Stopped 2024-09-15T02:51:07.420 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-15T02:51:07.420 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.490 INFO:tasks.ceph.osd.15:Stopped 2024-09-15T02:51:07.490 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T02:51:07.490 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-15T02:51:07.490 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.521 INFO:tasks.ceph.mgr.y:Stopped 2024-09-15T02:51:07.521 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T02:51:07.521 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.556 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T02:51:07.556 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T02:51:07.556 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T02:51:07.556 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.577 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T02:51:07.577 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T02:51:07.577 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.597 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T02:51:07.597 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T02:51:07.597 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~0s 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 is failed for ~0s 2024-09-15T02:51:07.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.15 is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-15T02:51:07.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-15T02:51:07.619 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T02:51:07.619 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T02:51:07.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-09-15T02:51:07.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:51:07.641 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T02:51:07.945 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:51:07.945 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T02:51:08.111 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:51:08.112 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-15T02:51:08.316 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:51:08.316 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-15T02:51:08.494 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:51:08.527 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T02:51:08.781 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:51:08.781 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T02:51:08.978 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:51:08.978 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-15T02:51:09.101 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:51:09.101 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-15T02:51:09.275 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:51:09.276 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T02:51:09.504 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:51:09.504 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-15T02:51:09.745 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:51:09.746 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-15T02:51:09.944 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:51:09.945 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-15T02:51:10.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:51:10.160 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T02:51:10.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:51:10.326 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-15T02:51:10.512 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:51:10.513 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-15T02:51:10.730 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:51:10.730 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-15T02:51:10.879 INFO:tasks.ceph:Archiving mon data... 2024-09-15T02:51:10.880 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/data/mon.a.tgz 2024-09-15T02:51:10.880 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-15T02:51:10.887 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.WQIvpn3hSv 2024-09-15T02:51:10.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WQIvpn3hSv 2024-09-15T02:51:11.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.WQIvpn3hSv 2024-09-15T02:51:11.067 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.WQIvpn3hSv is 437KB 2024-09-15T02:51:11.090 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.WQIvpn3hSv 2024-09-15T02:51:11.097 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/data/mon.b.tgz 2024-09-15T02:51:11.097 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-09-15T02:51:11.103 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.3am50tJCkE 2024-09-15T02:51:11.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3am50tJCkE 2024-09-15T02:51:11.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.3am50tJCkE 2024-09-15T02:51:11.276 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.3am50tJCkE is 623KB 2024-09-15T02:51:11.346 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.3am50tJCkE 2024-09-15T02:51:11.353 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/data/mon.c.tgz 2024-09-15T02:51:11.354 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-15T02:51:11.360 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.fqxntgoxxt 2024-09-15T02:51:11.361 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.fqxntgoxxt 2024-09-15T02:51:11.506 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.fqxntgoxxt 2024-09-15T02:51:11.578 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.fqxntgoxxt is 623KB 2024-09-15T02:51:11.606 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.fqxntgoxxt 2024-09-15T02:51:11.614 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T02:51:11.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:51:11.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:51:11.619 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:51:11.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T02:51:11.673 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T02:51:11.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:51:11.675 DEBUG:teuthology.orchestra.run.smithi118:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:51:11.678 DEBUG:teuthology.orchestra.run.smithi152:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:51:11.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T02:51:11.704 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:51:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:51:11.708 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:51:11.713 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T02:51:11.720 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T02:51:11.721 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi006/crash 2024-09-15T02:51:11.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:51:11.739 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi118/crash 2024-09-15T02:51:11.740 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:51:11.757 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi152/crash 2024-09-15T02:51:11.758 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:51:11.775 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi191/crash 2024-09-15T02:51:11.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T02:51:11.793 INFO:tasks.ceph:Compressing logs... 2024-09-15T02:51:11.793 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:51:11.795 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:51:11.797 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:51:11.815 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27532.log 2024-09-15T02:51:11.815 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31867.log 2024-09-15T02:51:11.815 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25731.log 2024-09-15T02:51:11.816 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27532.log: /var/log/ceph/ceph-client.admin.31867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31867.log.gz 2024-09-15T02:51:11.816 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.27532.log.gz-- 2024-09-15T02:51:11.816 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29642.log 2024-09-15T02:51:11.817 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26772.log 2024-09-15T02:51:11.817 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25731.log: /var/log/ceph/ceph-client.admin.29642.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.29642.log.gz 2024-09-15T02:51:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28122.log 2024-09-15T02:51:11.818 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25731.log.gz 2024-09-15T02:51:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31836.log 2024-09-15T02:51:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28274.log 2024-09-15T02:51:11.821 INFO:teuthology.orchestra.run.smithi006.stderr: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.26772.log.gz 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28122.log.gz 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31836.log.gz 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28821.log 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr: 28.4% -- replaced with /var/log/ceph/ceph-client.admin.28274.log.gz 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32264.log 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31303.log: /var/log/ceph/ceph-client.admin.28821.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz -- replaced with /var/log/ceph/ceph-client.admin.28821.log.gz 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30736.log 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28047.log 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30736.log: /var/log/ceph/ceph-client.admin.32264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30736.log.gz 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29810.log 2024-09-15T02:51:11.822 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32264.log.gz 2024-09-15T02:51:11.823 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33058.log 2024-09-15T02:51:11.823 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29810.log: /var/log/ceph/ceph-client.admin.28047.log: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.29810.log.gz 2024-09-15T02:51:11.823 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26161.log 2024-09-15T02:51:11.823 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28047.log.gz 2024-09-15T02:51:11.825 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33058.log.gz 2024-09-15T02:51:11.825 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28562.log 2024-09-15T02:51:11.825 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29191.log 2024-09-15T02:51:11.825 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26161.log.gz 2024-09-15T02:51:11.825 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28562.log.gz 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30763.log 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29330.log 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29191.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29191.log.gz 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30763.log: /var/log/ceph/ceph-client.admin.29330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32923.log 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30763.log.gz 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.29330.log.gz 2024-09-15T02:51:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26159.log 2024-09-15T02:51:11.827 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T02:51:11.829 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T02:51:11.830 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T02:51:11.830 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:51:11.830 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-15T02:51:11.830 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-15T02:51:11.830 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T02:51:11.833 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.9.log: 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:51:11.834 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:51:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27704.log 2024-09-15T02:51:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32923.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31856.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33382.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26159.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27704.log: /var/log/ceph/ceph-client.admin.31856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26138.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27704.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26196.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33382.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26138.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31414.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31249.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26196.log.gz 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26134.log 2024-09-15T02:51:11.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31414.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26148.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31249.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.26134.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30433.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26134.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26116.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26148.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30433.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31438.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26116.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25730.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31357.log 2024-09-15T02:51:11.838 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31438.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26730.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25730.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30154.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31357.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33085.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26730.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26775.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30154.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27588.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33085.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26563.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26775.log.gz 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.27588.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31114.log 2024-09-15T02:51:11.839 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27588.log.gz 2024-09-15T02:51:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31417.log 2024-09-15T02:51:11.840 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26563.log.gz 2024-09-15T02:51:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32572.log 2024-09-15T02:51:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31114.log.gz 2024-09-15T02:51:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33166.log 2024-09-15T02:51:11.840 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31417.log.gz 2024-09-15T02:51:11.841 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph.log: 84.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:51:11.867 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33805.log 2024-09-15T02:51:11.867 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.26581.log 2024-09-15T02:51:11.867 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.33805.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.26196.log 2024-09-15T02:51:11.867 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-15T02:51:11.867 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.26581.log: /var/log/ceph/bluestore_tool.26196.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-09-15T02:51:11.868 INFO:teuthology.orchestra.run.smithi152.stderr: 45.6% -- replaced with /var/log/ceph/bluestore_tool.26196.log.gz 2024-09-15T02:51:11.868 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.29609.log 2024-09-15T02:51:11.868 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/objectstore_tool..log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.25412.log 2024-09-15T02:51:11.868 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T02:51:11.868 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/objectstore_tool.29609.log: gzip -5 /var/log/ceph/objectstore_tool.25412.log: --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:51:11.868 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21939.log 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22315.log 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.21939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22048.log 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21939.log.gz 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21993.log 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22315.log.gz 2024-09-15T02:51:11.869 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22237.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22048.log.gz 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.21993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21993.log.gz 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22210.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22237.log: /var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22237.log.gz 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22129.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22210.log.gz 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22129.log: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.22129.log.gz 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22184.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22341.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.11.log: gzip /var/log/ceph/ceph-client.admin.22184.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.21965.log 2024-09-15T02:51:11.870 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22184.log.gz 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22290.log 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22341.log.gz 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.25844.log 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.21965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21965.log.gz 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22264.log 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22290.log.gz 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.25844.log: /var/log/ceph/ceph-osd.15.log 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr: 77.8% -- replaced with /var/log/ceph/ceph-client.0.25844.log.gz 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22156.log 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22264.log.gz 2024-09-15T02:51:11.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22075.log 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22100.log 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22156.log.gzgzip 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:-5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22075.log.gz 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.22100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22020.log 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22100.log.gz 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-15T02:51:11.872 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.22020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22020.log.gz 2024-09-15T02:51:11.875 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.audit.log: 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:51:11.875 INFO:teuthology.orchestra.run.smithi152.stderr: 93.7% -- replaced with /var/log/ceph/bluestore_tool.26581.log.gz 2024-09-15T02:51:11.875 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-15T02:51:11.875 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.26955.log 2024-09-15T02:51:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.6.log: 93.6% -- replaced with /var/log/ceph/bluestore_tool.33805.log.gz 2024-09-15T02:51:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.34187.log 2024-09-15T02:51:11.887 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.26955.log: 92.8% -- replaced with /var/log/ceph/bluestore_tool.26955.log.gz 2024-09-15T02:51:11.887 INFO:teuthology.orchestra.run.smithi152.stderr: 93.5% -- replaced with /var/log/ceph/objectstore_tool.25412.log.gz 2024-09-15T02:51:11.887 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30369.log 2024-09-15T02:51:11.895 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.34187.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33420.log 2024-09-15T02:51:11.895 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/objectstore_tool.30369.log: 93.8% -- replaced with /var/log/ceph/objectstore_tool.29609.log.gz 2024-09-15T02:51:11.895 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33424.log 2024-09-15T02:51:11.895 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.33420.log: 45.6% -- replaced with /var/log/ceph/bluestore_tool.33420.log.gz 2024-09-15T02:51:11.895 INFO:teuthology.orchestra.run.smithi152.stderr: 93.8% -- replaced with /var/log/ceph/bluestore_tool.34187.log.gz 2024-09-15T02:51:11.895 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:51:11.901 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.33424.log: 93.9% -- replaced with /var/log/ceph/bluestore_tool.33424.log.gz 2024-09-15T02:51:11.902 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28088.log 2024-09-15T02:51:11.902 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.26202.log 2024-09-15T02:51:11.902 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/objectstore_tool.28088.log: 84.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:51:11.902 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28849.log 2024-09-15T02:51:11.907 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T02:51:11.908 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/bluestore_tool.26202.log: 93.6% -- replaced with /var/log/ceph/bluestore_tool.26202.log.gz 2024-09-15T02:51:11.914 INFO:teuthology.orchestra.run.smithi152.stderr: 93.6% -- replaced with /var/log/ceph/objectstore_tool.30369.log.gz 2024-09-15T02:51:11.914 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32869.log 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32572.log: /var/log/ceph/ceph-client.admin.33166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32572.log.gz 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33166.log.gz 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29187.log 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25723.log 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29187.log: /var/log/ceph/ceph-client.admin.33139.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.29187.log.gz 2024-09-15T02:51:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26187.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29924.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25723.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32263.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26187.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28713.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.29924.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26536.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32263.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32316.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.28713.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32896.log 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26536.log.gz 2024-09-15T02:51:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25702.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32316.log.gz 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32896.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32896.log.gz 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28220.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25702.log.gz 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26920.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31429.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr: 22.1% -- replaced with /var/log/ceph/ceph-client.admin.28220.log.gz 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29897.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26920.log.gz 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33004.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31429.log.gz 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27965.log 2024-09-15T02:51:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29897.log: /var/log/ceph/ceph-client.admin.33004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29415.log 0.0% 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29897.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33004.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27446.log 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.27965.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26141.log 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29415.log: /var/log/ceph/ceph-client.admin.27446.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.29415.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27446.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25711.log 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27388.log 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26141.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.25711.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31440.log 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25711.log.gz 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33220.log 2024-09-15T02:51:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27388.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.27388.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29358.log 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31440.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31794.log 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33220.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32842.log 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29358.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17185.log 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31794.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.32842.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33436.log 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32842.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.17185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32293.log 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17185.log.gz 2024-09-15T02:51:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31803.log 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33436.log.gz 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32977.log 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32293.log.gz 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30466.log 0.0% 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31803.log.gz 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.32977.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27006.log 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32977.log.gz 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33463.log 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30466.log.gz 2024-09-15T02:51:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26698.log 2024-09-15T02:51:11.933 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27006.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27160.log 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33463.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26774.log 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26698.log: /var/log/ceph/ceph-client.admin.27160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26698.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27160.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31222.log 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33247.log 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26774.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27964.log 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31222.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32680.log 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33247.log.gz 2024-09-15T02:51:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27964.log.gz 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32680.log.gz 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30925.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25732.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29613.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30925.log: /var/log/ceph/ceph-client.admin.25732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30925.log.gz 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26103.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25732.log.gz 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26671.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29613.log.gz 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26105.log 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26103.log.gz 2024-09-15T02:51:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26671.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32815.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26105.log: 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26183.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26105.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32297.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32815.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.26183.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27339.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26183.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31428.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32297.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25725.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27339.log.gz 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28877.log 2024-09-15T02:51:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31428.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25725.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.25725.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30655.log 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32307.log 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28877.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.28877.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33490.log 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr: 25.6% -- replaced with /var/log/ceph/ceph-client.admin.30655.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28933.log 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32307.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26617.log 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33490.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29103.log 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28933.log.gz 2024-09-15T02:51:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30952.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26617.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29685.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29103.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.29103.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31087.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30952.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25721.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29685.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26779.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31087.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29077.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25721.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30407.log 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26779.log.gz 2024-09-15T02:51:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31873.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29077.log.gz 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30979.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30407.log.gz 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 /var/log/ceph/ceph-client.admin.31873.log: --verbose -- /var/log/ceph/ceph-client.admin.29160.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31873.log.gz 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25618.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30979.log.gz 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29160.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29160.log.gz-5 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:--verbose -- /var/log/ceph/ceph-client.admin.31857.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31427.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25618.log.gz 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33571.log 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2024-09-15T02:51:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27472.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31427.log.gz 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31412.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33571.log.gz 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28513.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.27472.log.gz 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31412.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31412.log.gz 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26108.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28513.log.gz 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26921.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32653.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26108.log.gz 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31141.log 2024-09-15T02:51:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26921.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30817.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32653.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.31141.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26129.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31141.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31423.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30817.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30380.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26129.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28305.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31423.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31861.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30380.log.gz 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25521.log 2024-09-15T02:51:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28305.log.gz 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25716.log 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31861.log.gz 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33625.log 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25521.log.gz 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27789.log 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25716.log.gz 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32252.log 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33625.log.gz 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32734.log 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27789.log.gz 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32761.log 2024-09-15T02:51:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32252.log.gz 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33544.log 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32734.log.gz 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30871.log 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32761.log.gz 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26111.log 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33544.log.gz 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33517.log 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30871.log.gz 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26140.log 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26111.log.gz 2024-09-15T02:51:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31875.log 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33517.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33517.log.gz-- 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.22135.log 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26140.log.gz 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.31875.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31787.log 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31875.log.gz 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27216.log 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.22135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22135.log.gz 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28963.log 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31787.log.gz 2024-09-15T02:51:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27244.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27216.log.gz 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25495.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.28963.log.gz 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.27244.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33193.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.27244.log.gz 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30898.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25495.log.gz 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21992.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33193.log.gz 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30069.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30898.log.gz 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.21992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33598.log 2024-09-15T02:51:11.945 INFO:teuthology.orchestra.run.smithi006.stderr: 28.4% -- replaced with /var/log/ceph/ceph-client.admin.21992.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25729.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr: 22.0% -- replaced with /var/log/ceph/ceph-client.admin.30069.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.33598.log: -- /var/log/ceph/ceph-client.admin.27729.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33598.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33031.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25729.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33409.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27729.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.27729.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28075.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33031.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32950.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33409.log.gz 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.28075.log: -- /var/log/ceph/ceph-client.admin.31060.log 2024-09-15T02:51:11.946 INFO:teuthology.orchestra.run.smithi006.stderr: 22.9% -- replaced with /var/log/ceph/ceph-client.admin.28075.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31444.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32950.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30074.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31060.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33274.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31444.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32267.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30074.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28360.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33274.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27095.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32267.log.gz 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25708.log 2024-09-15T02:51:11.947 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28360.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30493.log 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27095.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22277.log 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25708.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31409.log 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30493.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.22277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30273.log 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22277.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.31409.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29979.log 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31409.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25722.log 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30273.log.gz 2024-09-15T02:51:11.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29785.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29979.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25722.log: /var/log/ceph/ceph-client.admin.28451.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25722.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32248.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29785.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31435.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28451.log: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.28451.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27867.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32248.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.31435.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25645.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31435.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27867.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.27867.log.gz 2024-09-15T02:51:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27011.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25645.log.gz 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32518.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31839.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27011.log.gz 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31825.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32518.log: /var/log/ceph/ceph-client.admin.31839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30628.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32518.log.gz 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31839.log.gz 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32788.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31825.log.gz 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30628.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T02:51:11.950 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30628.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr: 84.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31835.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32788.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30709.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31276.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31835.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26644.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30709.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.31276.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29838.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31276.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25591.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26644.log.gz 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30601.log 2024-09-15T02:51:11.951 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29838.log.gz 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30682.log 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25591.log.gz 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30601.log.gz 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30207.log 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30682.log.gz 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31006.log 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.30207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31168.log 2024-09-15T02:51:11.952 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30207.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31436.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31006.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28650.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31168.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33355.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31436.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.28650.log: -- /var/log/ceph/ceph-client.admin.32626.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28650.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25710.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33355.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32626.log.gz 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30211.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32545.log 2024-09-15T02:51:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25710.log.gz 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25547.log 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30211.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.30211.log.gz 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31195.log 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32545.log.gz 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30547.log 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25547.log.gz 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29470.log 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31195.log.gz 2024-09-15T02:51:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30547.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30547.log.gz 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29470.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.29470.log.gz-5 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:--verbose -- /var/log/ceph/ceph-client.admin.28796.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27638.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.8.log: /var/log/ceph/ceph-client.admin.28796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31441.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28796.log.gz 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33301.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.27638.log.gz 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29558.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31441.log.gz 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26590.log 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33301.log.gz 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29558.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.29558.log.gz 2024-09-15T02:51:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27821.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31858.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.26590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26590.log.gz 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27821.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31033.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27821.log.gz 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31858.log.gz 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32287.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28509.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31033.log.gz 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32287.log.gz 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28592.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28509.log.gz 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28848.log 2024-09-15T02:51:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31439.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28592.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.28592.log.gz 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31872.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30790.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31439.log: gzip 0.0%-5 --verbose -- /var/log/ceph/ceph-client.admin.30181.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31439.log.gz 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31872.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32707.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30790.log: 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30790.log.gz 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30181.log.gz 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33650.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25694.log 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32707.log.gz 2024-09-15T02:51:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33650.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25728.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33650.log.gz 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25694.log.gz 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32599.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33112.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25728.log.gz 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32599.log.gz 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33328.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30520.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33112.log.gz 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31925.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33328.log.gz 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30574.log 2024-09-15T02:51:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30520.log.gz/var/log/ceph/ceph-client.admin.31925.log: 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31925.log.gz 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25720.log 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30574.log.gz 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/ceph-client.admin.31330.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gz 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25720.log.gz 2024-09-15T02:51:11.959 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-09-15T02:51:11.960 INFO:teuthology.orchestra.run.smithi118.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T02:51:11.960 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/objectstore_tool.28849.log: /var/log/ceph/ceph-osd.2.log: 93.8% -- replaced with /var/log/ceph/objectstore_tool.28849.log.gz 2024-09-15T02:51:11.960 INFO:teuthology.orchestra.run.smithi152.stderr: 93.7% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-09-15T02:51:11.960 INFO:teuthology.orchestra.run.smithi152.stderr: 94.2% -- replaced with /var/log/ceph/objectstore_tool.28088.log.gz 2024-09-15T02:51:11.984 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T02:51:12.149 INFO:teuthology.orchestra.run.smithi006.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T02:51:12.171 INFO:teuthology.orchestra.run.smithi152.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-15T02:51:12.176 INFO:teuthology.orchestra.run.smithi118.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T02:51:12.210 INFO:teuthology.orchestra.run.smithi118.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-15T02:51:12.271 INFO:teuthology.orchestra.run.smithi191.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T02:51:12.282 INFO:teuthology.orchestra.run.smithi006.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-15T02:51:12.291 INFO:teuthology.orchestra.run.smithi152.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-15T02:51:12.304 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-15T02:51:12.307 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T02:51:12.320 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-15T02:51:12.342 INFO:teuthology.orchestra.run.smithi006.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T02:51:12.366 INFO:teuthology.orchestra.run.smithi006.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-15T02:51:12.395 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-15T02:51:12.446 INFO:teuthology.orchestra.run.smithi152.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T02:51:12.446 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-15T02:51:12.447 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m0.619s 2024-09-15T02:51:12.447 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m2.038s 2024-09-15T02:51:12.447 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.096s 2024-09-15T02:51:12.472 INFO:teuthology.orchestra.run.smithi006.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T02:51:12.473 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-15T02:51:12.473 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m0.675s 2024-09-15T02:51:12.473 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m2.391s 2024-09-15T02:51:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.135s 2024-09-15T02:51:12.599 INFO:teuthology.orchestra.run.smithi118.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-15T02:51:12.601 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-09-15T02:51:12.601 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m0.800s 2024-09-15T02:51:12.601 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m2.043s 2024-09-15T02:51:12.601 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.090s 2024-09-15T02:51:12.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.15 has been restored 2024-09-15T02:51:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-15T02:51:12.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-15T02:51:12.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-15T02:51:12.628 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-15T02:51:12.628 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-15T02:51:12.628 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m0.792s 2024-09-15T02:51:12.628 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m2.199s 2024-09-15T02:51:12.628 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.074s 2024-09-15T02:51:12.629 INFO:tasks.ceph:Archiving logs... 2024-09-15T02:51:12.629 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi006/log 2024-09-15T02:51:12.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:51:13.198 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi118/log 2024-09-15T02:51:13.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:51:13.566 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi152/log 2024-09-15T02:51:13.567 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:51:14.603 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi191/log 2024-09-15T02:51:14.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T02:51:15.048 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T02:51:15.058 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T02:51:15.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:51:15.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:51:15.064 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:51:15.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T02:51:15.179 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T02:51:15.189 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T02:51:15.189 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:51:15.191 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:51:15.193 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:51:15.196 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:51:15.210 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:51:15.210 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-09-15T02:51:15.210 INFO:teuthology.orchestra.run.smithi006.stdout:+hv01.front.sepi 67.205.162.81 3 u 41 64 77 0.088 0.047 1.189 2024-09-15T02:51:15.211 INFO:teuthology.orchestra.run.smithi006.stdout:+hv02.front.sepi 63.231.80.2 3 u 45 64 77 0.105 0.571 1.191 2024-09-15T02:51:15.211 INFO:teuthology.orchestra.run.smithi006.stdout:*hv03.front.sepi 74.6.168.72 3 u 46 64 77 0.113 0.373 1.195 2024-09-15T02:51:15.211 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:51:15.214 INFO:teuthology.orchestra.run.smithi152.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:51:15.215 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================== 2024-09-15T02:51:15.215 INFO:teuthology.orchestra.run.smithi152.stdout:+hv01.front.sepi 67.205.162.81 3 u 45 64 77 0.099 0.012 1.035 2024-09-15T02:51:15.215 INFO:teuthology.orchestra.run.smithi152.stdout:+hv02.front.sepi 63.231.80.2 3 u 45 64 77 0.206 0.473 1.067 2024-09-15T02:51:15.215 INFO:teuthology.orchestra.run.smithi152.stdout:*hv03.front.sepi 74.6.168.72 3 u 50 64 77 0.143 0.351 1.237 2024-09-15T02:51:15.215 INFO:teuthology.orchestra.run.smithi152.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:51:15.217 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:51:15.217 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2024-09-15T02:51:15.217 INFO:teuthology.orchestra.run.smithi118.stdout: hv01.front.sepi 67.205.162.81 3 u 44 64 77 0.083 -0.463 0.104 2024-09-15T02:51:15.217 INFO:teuthology.orchestra.run.smithi118.stdout: hv02.front.sepi 63.231.80.2 3 u 42 64 77 0.087 0.060 0.072 2024-09-15T02:51:15.217 INFO:teuthology.orchestra.run.smithi118.stdout:*hv03.front.sepi 74.6.168.72 3 u 46 64 77 0.098 -0.063 0.076 2024-09-15T02:51:15.217 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:51:15.219 INFO:teuthology.orchestra.run.smithi191.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:51:15.220 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================== 2024-09-15T02:51:15.220 INFO:teuthology.orchestra.run.smithi191.stdout:+hv01.front.sepi 67.205.162.81 3 u 43 64 77 0.069 -0.271 0.538 2024-09-15T02:51:15.220 INFO:teuthology.orchestra.run.smithi191.stdout:+hv02.front.sepi 63.231.80.2 3 u 46 64 77 0.060 1.156 0.690 2024-09-15T02:51:15.220 INFO:teuthology.orchestra.run.smithi191.stdout:*hv03.front.sepi 74.6.168.72 3 u 48 64 77 0.102 0.156 0.458 2024-09-15T02:51:15.220 INFO:teuthology.orchestra.run.smithi191.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:51:15.220 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T02:51:15.230 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T02:51:15.230 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T02:51:15.239 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T02:51:15.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T02:51:15.260 INFO:teuthology.task.internal:Duration was 727.977322 seconds 2024-09-15T02:51:15.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T02:51:15.270 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T02:51:15.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:51:15.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:51:15.275 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:51:15.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T02:51:15.329 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T02:51:15.329 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-09-15T02:51:15.330 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:51:15.342 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-09-15T02:51:15.342 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:51:15.357 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-09-15T02:51:15.357 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:51:15.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-09-15T02:51:15.373 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T02:51:15.390 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T02:51:15.390 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:51:15.392 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:51:15.401 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:51:15.404 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T02:51:15.443 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T02:51:15.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:51:15.481 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:51:15.483 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:51:15.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T02:51:15.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T02:51:15.574 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T02:51:15.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:51:15.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:51:15.607 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:51:15.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T02:51:15.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T02:51:15.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.664 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.673 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-09-15T02:51:15.676 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-09-15T02:51:15.678 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-09-15T02:51:15.699 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-09-15T02:51:15.729 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:51:15.761 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:51:15.768 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:51:15.774 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:51:15.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T02:51:15.789 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T02:51:15.790 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi006 2024-09-15T02:51:15.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:51:15.819 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi118 2024-09-15T02:51:15.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:51:15.846 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi152 2024-09-15T02:51:15.847 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:51:15.874 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905171/remote/smithi191 2024-09-15T02:51:15.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T02:51:15.904 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T02:51:15.904 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:51:15.907 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:51:15.910 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:51:15.913 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T02:51:15.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T02:51:15.930 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T02:51:15.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T02:51:15.940 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T02:51:15.941 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:51:15.956 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:51:15.959 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:51:15.961 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T02:51:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:51 /home/ubuntu/cephtest 2024-09-15T02:51:15.964 INFO:teuthology.orchestra.run.smithi118.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:51 /home/ubuntu/cephtest 2024-09-15T02:51:15.967 INFO:teuthology.orchestra.run.smithi152.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:51 /home/ubuntu/cephtest 2024-09-15T02:51:15.969 INFO:teuthology.orchestra.run.smithi191.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 02:51 /home/ubuntu/cephtest 2024-09-15T02:51:15.972 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T02:51:15.981 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T02:51:16.008 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 727.977322101593 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T02:51:16.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:51:16.092 INFO:teuthology.run:pass